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ABSTRACT 


Human performance in spatial orientation tasks is mainly determined by spatial 
awareness and the skills to transition from the current spatial attitude into the desired 
spatial orientation and position. Erroneous spatial awareness may lead to degraded task 
performance, loss of equipment, serious injuries, or fatal aviation mishaps. 

This study investigated human orientation performance in relation to display 
designs that support mental models of the user’s spatial situation. The goal of this study 
was to develop, design, implement, and test a prototype of an instrument to support 
operators in extreme flight tasks by a pictorial avatar design. 

Two experiments have been conducted during this study to investigate how 
the proposed design affects operators' orientation performance and to prove the 
design concept. The most important results can be summarized as follows: 

1. The proposed pictorial avatar design significantly reduced the time to assess the 
spatial situation. 

2. Performance, in terms of recovery time and precision, is significantly improved by 
the proposed design compared to traditional flight instrumentation. 

3. The proposed design was intuitively accepted, interpreted, and used to solve 3D- 
orientation tasks efficiently. 

4. The proposed design reduced greatly extreme misjudgments of the own spatial 
orientation. 
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I. INTRODUCTION 


The design of the instrument display panels in the eoekpits of today’s aireraft has 
a long history, beginning over a eentury ago. From the early primitive instruments of 
flight pioneers to the modem multifunctional design, the cockpit display has been the 
pilot and flight crew’s sole direct source of information and continues to be an invaluable 
tool in the prevention of aircraft accidents. 

Research shows that a large number of aircraft accidents are due to the pilot’s 
spatial disorientation and that many of those accidents may be due to improper 
instmment interpretation and use (Clay, 1993). The need to improve in those cmcial areas 
and the developing need to fly without visual references were the incentives that drove 
flight engineers over the last century to gradually improve the design of flight 
instmments (Previc & Ercoline, 2004). Technological breakthroughs enabled them to 
develop multifunctional instmments and the so-called glass cockpit (Tsang & Vidulich, 
2003). Moreover, there is a growing demand to make more information available to the 
pilot. Thus both the number and the complexity of the instmment displays have 
increased, and the limits of human information-processing resources may now be 
exceeded by the overwhelming increase in warning indicators, status displays, flight path 
displays, air-traffic-control data links, meteorological information, navigational 
information, and communications data (Mejdal, McCauley, & Beringer, 2001). As a 
result, the requirement for human-centered design of displays has also become 
increasingly important. In addition, more and more aircraft are remotely controlled by 
operators on the ground who may have no background training as pilots. For these 
reasons, cockpit instmments must become less abstract and more intuitively 
understandable than the conventional or traditional designs. 

The use of instmments in aviation is very different from that of the common 
human-computer interaction. Since a pilot or a remote-control aircraft operator acts in a 
highly dynamic environment, their need to quickly and accurately access information has 
to be addressed by a human factor-based display design (Temme, 2004). 
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In light of all the above factors, the main motivation for this thesis was the 
extreme impact of spatial disorientation (SD) on aviation accidents. SD is now a major 
issue in modern fixed-wing and rotor-wing aviation accident prevention (Tsang & 
Vidulich, 2003). Accidents due to spatial disorientation result in nearly three times more 
deaths than non-SD accidents (Previc & Ercoline, 2004). 

According to the U.S. Air Force Safety Center, the rate of USAF SD-related 
accidents per 100,000 flying hours changed little from 1991 to 2000. Moreover, analysis 
of the twenty years prior reveals a similar rate. Data from the U.S. Army and Navy and 
the Federal Aviation Administration (FAA) shows similar findings (Previc & Frcoline, 
2004). Other studies show that, when operating unmanned aerial vehicles, the need to 
maintain spatial awareness and to prevent spatial disorientation is becoming a vital issue 
(McCauley & Matsangas, 2004). 

This study investigated human orientation performances in relation to display 
designs that support a user’s mental picture of a spatial situation. Previous research on 
egocentric and exocentric instrument designs identified a performance degradation by 
misinterpreting abstract 2D- and 3D-displays in situations of high workload, fatigue, or 
stress (Fndsley, M. R., 1999b). A proposed solution is a pictorial avatar design that 
implements major design principles for a human-centered-system interface (Wickens, 
Fee, Fiu, & Becker, 2004). Two tasks in a flight simulation were investigated to 
determine how the proposed design affected operators’ orientation performance and to 
prove or disprove the new design concept. 
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II. BACKGROUND 


A, SPATIAL AWARENESS 

1. Models of Situation Awareness 

To better understand the term “spatial awareness,” we must define it and other 
terms used in similar eontexts. “Spatial awareness” is one aspeet of situational awareness 
(SA), whieh is eommonly defined as “being aware of what is happening around you and 
understanding what information means to you now and in the future” (Endsley, M.R., 
Bolte, & Jones, 2003). “Situational Awareness” is more formally defined as “the 
perception of the elements in the environment within a volume of time and space, the 
comprehension of their meaning, and the projection of their status in the near future” 
(Endsley, M.R., 1988). Scientists distinguish between personal, or individual, SA and 
team-based, or shared, SA (Cannon-Bowers, Tannenbaum, Salas, & Volpe, 1995). In 
summarizing the relevant SA-related literature, Cannon-Bowers makes the following 
important points about the nature of SA: 

1. Mental models are important for individual SA. 

2. Shared mental models are important to team performance because they allow 
team members to form the necessary explanations and expectations of team 
and task actions. 

3. Team (shared) SA is important to team performance. 

In this study we will be concerned solely with individual SA. 

Endsley (2003) discusses three levels, or degrees, of SA: (1) perception of the 
elements in the environment, (2) comprehension of the current situation, and (3) 
projection of future status. In the first degree of SA, the perception of the elements in the 
environment, humans achieve SA by perceiving the status, characteristics, and dynamics 
of all relevant elements in the environment. Eor instance, pilots must perceive important 
elements, such as the status of their own aircraft, other air traffic, terrain, flight 
parameters, and warnings. In military aviation, the number of tasks and the rate of change 
in the environment is very high because of the dynamic flight maneuvers, ground- and 
airborne-enemy activities, sophisticated weapon systems, etc. 
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Endsley’s second level of SA, comprehension of the current situation, is based on 
synthesis Level 1 elements. The key for Level 2 SA is that the human must integrate it 
with Level 1 information to form a complete mental representation of the environment. 
This includes assigning levels of significance to objects or information. 

The third level of SA, the projection of future status, is the highest level of SA. It 
enables a person to predict, at least for the immediate future, the situation’s 
characteristics. To achieve this, the pilot must have a thorough understanding of the 
system’s functions and dynamics and complete comprehension of the situation (i.e., both 
Level 1 and Level 2 SA). The pilot must constantly project his current situation into the 
future and base his decisions on that prediction. 

Another approach is Miller and Shattuck’s (2004) “situated cognition model”, 
which describes SA as a process rather than certain fixed states as in Endsley’s (1999a) 
model. It is based, on the one hand, on Pew’s model of ideal SA, achievable SA, and 
actual SA (Pew, R.W., 2000), which describes the problem of measuring SA. On the 
other hand. Miller’s model includes Tucker and Hammond’s lens-model of information 
selection and filtering (Tucker, 1964). Thus, the Miller and Shattuck’s model focuses on 
processes rather than states and includes both the human and the machine elements of a 
system. It is oriented toward assessing human-system performance, tracks the evolution 
of activities and cognition, and links it to cognitive decision processes. Prom that 
perspective, the model describes the process of building SA starting at the ground truth, 
which is selectively perceived by different layers of sensors (lenses). Every lens filters 
and transforms the received information, which is then processed by methods similar to 
Endsley’s three-levels-of-SA model. Nevertheless, in Miller and Shattuck’s model, those 
levels are not independent levels of SA. They are specific cognitive processes that map 
additional aspects of the task and the environment - such as goals, expectations, 
personality, etc. - to the originally perceived information (Miller & Shattuck, 2004). 
Originally developed to explain decision-making processes in military command and 
control environments, this model is easily applicable for other cases of situation 
awareness. 


4 



Research shows that it is common for cognitive models to focus on processes of 
information processing and selection. We find that the process of active search for 
additional information is underrepresented or only vaguely explained by feedback loops. 
A more complete model would address the active-search process at the same level, or 
with the same degree, of accuracy as the information processing and selection processes. 
When a set of perceived information is not sufficient to build a desired cognitive picture 
of the current level of SA, humans start to actively search for additional information. A 
pilot will conduct this search according to certain patterns, derived from his training and 
experience and based on the importance of the information and the ease of access to it. 
For instance, a pilot who has the impression that his airplane is loosing power will 
actively check other indicators, like instruments, warnings, or outside cues, to complete 
and verify his first inkling of SA. He might try flight maneuvers or different control 
configurations to narrow the problem or to determine the seriousness of the problem. 

In sum, we understand the process of building situation awareness as an active 
cognitive problem-solving process. Hence, a design that supports and enhances SA must 
pay attention to multiple loops of information processing that support the active nature of 
gaining personal SA. 

2. Models of Spatial Awareness 

According to the common definition of SA, in a three-dimensional (3D) 
orientation task, a person who has good SA has rapid access to an accurate mental picture 
of the changing environment and may be able to predict what his spatial situation will be 
in the near future. For example, though an aviator with excellent situation awareness may 
not be actively thinking about the fact that there is an aircraft nearby, he will be able to 
operate the aircraft proactively and appropriately, as well as quickly and precisely, 
because of his ability to rapidly access the necessary information from memory, because 
he has a mental picture of the current spatial situation (Wickens, 2002). Thus, designs of 
flight instrument that support SA will facilitate appropriate reactions in uncertain 
situations. The above discussion illustrate what is meant by the term “spatial awareness.” 
Spatial awareness can be seen as a subset of situation awareness or, more specifically, as 
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an application of SA. The relevant literature shows many other, often vague, definitions. 
In this study, spatial awareness is considered as an extension of spatial orientation, where 
spatial orientation is understood as the ability to reeognize an objeet’s orientation or 
position in three-dimensional spaee. In that defined eontext, spatial awareness is a pilot’s 
knowledge of his own aireraft’s spatial attitude and position relative to a desired flight 
route and the ground. In addition, spatial awareness ineludes the pilot’s knowledge about 
his eurrent flight maneuver, his own plane’s spatial behavior, and the spatial behavior of 
other air traffie in dynamie situations. We intentionally exelude other faetors like the 
status of the erew, the status of the aireraft, etc. Therefore, our definition follows the 
argument made in a NASA flight-instrument study that describes “spatial awareness” as 
meaning that a pilot is increasingly able to think ahead of the aireraft and ean do this for a 
variety of future situations. Thus, in NASA’s deseription, the pilot’s spatial awareness 
involves his fundamental knowledge about both his eurrent state and antieipated future 
goals (Parrish, R.V., Busquets, Williams, & Nold, 1994). 

Wiekens (2002) points out that a pilot’s spatial awareness is determined by six 
erueial variables: the piteh, roll, and yaw (slip) of the aireraft, its altitude, its deviation 
from a designated flight path, and its position along its eurrent flight path. Additional 
faetors, however, must be eontinuously monitored, both direetly and indireetly, to build a 
eomplete spatial model. Two of the most obvious faetors are the air speed and the stall- 
indieator. Knowledge about other air traffic, the strueture of the ground, and the weather 
is also neeessary. All of these variables are eross-linked, whieh means that one or more of 
them will infiuenee other variables in the future. For instance, the eombination of piteh 
and airspeed may eventually eause the aircraft to stall. Skilled pilots represent this 
linkage in a mental model, also referred to as “mental pieture” or “mental image”, and 
implement time eonstrains, lateneies, and additional information within it. 

3. Measuring SA and Spatial Awareness 

From our perspeetive, it is not enough to argue how we ean infiuenee SA and 
spatial awareness; we must also discuss the means available for measuring the eonstruct 
itself Sinee we defined “spatial awareness” as a subset of SA, we infer that we ean 
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measure spatial awareness by applying prineiples of SA measurement. An individual’s 
SA is a relatively diffieult eonstruet to measure (Haneoek & Desmond, 2001). This is 
partieularly true for spatial awareness, sinee it is inherently dynamie. 

Another issue to determine is how mueh SA is suffieient for a given task. 
Endsley, Sollenberger, and Stein (2000) find that SA and performanee measures are only 
linked probabilistieally, and that there are no set thresholds of SA to guarantee a given 
level of performanee. Therefore they introdueed the term “relative levels of SA” to 
deseribe the relative ehange of SA prior and after the ehange of a relevant faetor of SA. 
Measures of SA should also eonsider the “ideal spatial awareness”, that is, perfeet 
knowledge of all relevant aspeets of a situation, with no gaps or holes. As noted earlier. 
Pew (2000) states that the aetual SA tends to be a subset of the aehievable SA, whieh is 
itself a subset of the ideal SA of any situation. Given the eomplexities of the real world, 
perfeet SA is generally not feasible. 

An individual’s level of SA will be determined partially by the quantity and 
quality of the information available and partially by the individual’s ability to use 
important information sourees to fill in any missing, impreeise, or ineomplete 
information. An individual will have a low level of situation awareness if he fails to 
antieipate vital faetors due to his laek of awareness of their importanee, or if his attention 
is foeused on other things (Entin, 1998). Entin (1998) eonsiders two types of SA 
measures: high-level SA and detailed SA. High-level SA measures faets aeeording to 
subjeets’ responses to general questions about an evolving situation. A detailed SA 
measure is based on questions about the elements of a situation in a way similar to the 
measures derived from the Situation Awareness Global Assessment Teehnique 
(SAGAT), whieh is based on a broad-ranging assessment of operator SA requirements 
(Gawron, 2000) . 

Entin (1998) eondueted researeh on SA during military operations. He suggests 

that detailed SA is more integrated and less ambiguous during phases in whieh the 

situation at hand is more statie and time is not pereeived to be a eritieal faetor (e.g., the 

planning phase). Detailed SA degrades when ineoming information ehanges rapidly. In 

aviation tasks, operators may not have time to integrate all the ineoming information into 
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a coherent picture. Henee, the level of detailed SA will be less eonsistent in highly 
dynamic situations. High-level SA, by eontrast, does not evidence that pattern. Rather, it 
reflects the notion of a strong relationship between the levels of SA in adjaeent phases, 
whieh ean be explained by differenees in the eognitive proeesses operational in the two 
types of situational awareness. High-level SA may depend more on the aeeuraey of an 
individual’s mental model of a situation, while detailed SA may depend more on the 
individual’s pereeptual skills, memory, and ability to extraet and eombine speeifie bits of 
information. 

Entin’s study of detailed and high-level SA produeed two basie types of SA 
measures. One is based on subjeets’ responses to questionnaires that ineluded questions 
designed to eapture speeifie elements of a given situation. These researeh methods are 
measures of detailed SA. Subjeet matter experts (SMEs) ean be used to determine the 
precision required for a eorreet response. Measures of high-level SA ean be gathered by 
administering open-ended questions about aspeets of an overall situation. 

Aceording to Pew (2000), there are several other means by whieh SA ean be 
measured, whieh fall into three eategories: direct systems performance measures, verbal 
protocols and subjective measures. Direct systems performance measures are only 
applieable in very limited number of situations. However, disruptions intended to 
disorient the operator ean be introduced (Pew, R. W. & Mavor, 1998), and the operator’s 
reeovery time and the suecess of the reeovery tacties ean then be measured. Pure 
performanee measures are often not suffieient to diagnose human-system relationships in 
detail. In many eases we have to measure subjeetive issues sueh as mental models or the 
mental workload as well (Endsley, M.R. et ah, 2003). 

Condueting verbal protocols involves the direet reeording of observers’ or 
subjeets’ verbal information during or immediately after an exereise or from videotape 
reeords of the experiment (Adelman, 2003). Here, subjeets may be asked to think aloud 
or explain information as they work. This method is most valuable at the beginning of an 
evaluation when eoncepts are still being refined. However, a disadvantage is that 
thinking aloud may change the cognitive process. In addition, the observations may not 
be free from subjective interpretations. 
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Subjective measures include self-assessments, expert judgments, peer ratings, and 
supervisor or instructor ratings (Matthews, Davies, Westerman, & Stammers, 2000). The 
best of these teehniques in regard to the highest validity appears to be the self- 
administered SA test, the Situation Awareness Rating Teehnique (SART), whieh involves 
a three-seale subset, the results of whieh are eombined via algebraic equation to produee 
an overall estimate of the subjeet’s SA (Endsley, M.R., Seleon, Hardiman, & Croft, 
1998). 

Endsley et al. (1998) deseribe the outcome of a study eonducted to eompare the 
SART and SAGAT measures of SA using twelve experieneed RAE pilots engaged in a 
simulated C2 environment. SART is a popular SA measurement tool beeause it is easy to 
use and to administer. Another advantage of this test is that it ean be used for a wide 
range of tasks, ineluding real-world tasks as well as simulations, without changing the 
domains of the scales. However, SART does have some limiting characteristics. Endsley 
(1998) ineludes these three: 

(1) The laek of operators’ ability to judge their own SA. 

(2) The potential influenee of performanee on their ratings. 

(3) The possible eonfounding with workload issues, though SA may operate as a 
factor independent from workload in some situations. 

The study shows that SART and SAGAT measures of SA are not eorrelated with 
one another. Eurthermore, the study shows that, despite the stated limitations, SART is 
applieable to prediet the outeome in aetual flight situations when detailed performanee 
measures are not available. However, SAGAT is able to reveal more hidden interaetions 
of factors and presents a higher level of information. A disadvantage is that SAGAT is 
not applieable for in-flight evaluations and needs mueh more data to gain statistical 
significance in comparison to SART. 

Another problem with SAGAT is that often the situation has been frozen at some 
moment to evaluate the current SA. This interrupts the natural flow of the task and makes 
the whole situation somewhat artificial. The question that arises is whether these 
assessments are valid. In one study, real-time probes (i.e., SAGAT questions during 
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ongoing operations) and stopped-system probes (SAGAT, SART, and NASA-TLX 
[NASA’s workload survey]) were eondueted to evaluate an operator’s SA in one 
scenario. The results were not clear, but they did indicate that real-time probes do 
measure SA at some level. Further research was recommended (Jones & Endsley, 2000). 

A different way to measure the gradient of change in SA is to infer it from 
objective performance measurements. Endsley (2003) concluded that the change was 
caused by a change in the individual’s SA, if an appropriate set of measurable parameters 
is chosen and measured. In our opinion, an objective performance measurement should 
be conducted wherever possible. Subjective methods should be conducted to explain and 
diagnose data from objective measurements. 

Process measures are considered very effective in measuring SA. These may 
include eye-tracing, head tracking, information acquisition, flight parameter analysis, and 
so on (Endsley, M.R. et ah, 2003). 

B, HUMAN PERFORMANCE AND COGNITION 

1, Human Perception and Cognition 

Throughout the scientific literature there exists a wide variety of models of human 
perception and cognition. The term “cognition” is used in several ways. In psychology 
cognition refers to the mental processes of an individual, with particular emphasis on a 
view that argues that the mind has internal mental states (such as beliefs, desires, and 
intentions). Cognition can be understood in terms of information processing, especially 
when a lot of abstraction is involved, or processes such as knowledge, expertise, or 
learning, for example, are at work. The term “cognition” also is used in a wider sense to 
mean the act of knowing or knowledge. It derived from psychological science which 
attempted to explain human behavior and reasoning (Matthews et ah, 2000). Unlike 
behaviorism, cognitive theory focuses on what is going on inside the person’s mind. 
Cognitive learning is not just a change in behavior; it is a change in the way a person 
thinks, understands, or feels. 

“Perception” is defined as a direct consequence of selective attention, which 
involves the extraction of meaning from a set of information processed by the human 
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senses (Wiekens et al, 2004). In this study we foeus on higher-level pereeptions whieh 
answer questions sueh as “Where I am?”, “What is my spatial orientation?”, “Where am I 
going?”, and so on (Warren & Wertheim, 1990). Hence, we start with thinking at a level 
above human sensory systems and the physical stimulus. 

Wiekens et al. (2004) distinguish three basic perceptual processes: (1) bottom-up 
feature analysis, (2) unitization, and (3) top-down processing. Bottom-up feature analysis 
is done before the given information is passed to the memory for “higher”-level 
processing. However, the basic idea is that patterns can be described in terms of their 
more basic features. Experience helps in selecting necessary information out of the 
complex stream of features. The process of information selection to determine if events 
occurred together in the past and if they build a pattern is known as utilization. Wiekens 
et al. (2003) point out that successful utilization accelerates and automates the perception 
process. If a certain pattern is recognized, humans subconsciously tend to solve the 
perceived information puzzle by replacing an inhomogeneous set of information with a 
previously built meaning. Wiekens (1995) uses the example of a well-known word. 
Humans usually recognize the whole word immediately, rather then processing all the 
letters and then making sense of the word. The opposite is the case if the human is 
confronted with a totally unknown word. Association and context build a frame for 
perceptional pattern-matching and are driven by expectations and past experience. This is 
known as top-down processing, in which humans are able to guess the correct meaning 
even when the information is incomplete or unclear. 

To explain cognitive models, we must first think about a cognitive architecture. It 
is hard, if not impossible, to measure cognitive processes directly. We are most often 
limited to comparing and interpreting the input and output stages of human information¬ 
processing systems. Endsley et al. (2003) provide a model of how perception, cognition, 
SA, decisions, and performance may be connected (Eigure 1). 
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Figure 1. Evaluation metrics at different stages of user performance 
(From: Endsley et al., 2003, p. 58) 


However, researchers distinguish between objective and subjective assessments of 
human cognition. Introspective evaluation methods may deliver valid representations of 
the inner cognitive world of a person, but lack the strong objective requirements of 
experimental methods. Many experiments have shown that unconscious biases 
significantly influence experimental outcomes (Wickens et ah, 2004). This is true even 
when the questioned person feels confident that his reports are accurate. Due to these 
problems, researchers try to find information-processing models that increase the 
objectiveness of their measures (Rosenthal, 2003). Matthews (2000) describes one 
attempt to create computational models that predict the performance of a person in the 
real world. Even with simple tasks it was shown that these models have various 
weaknesses. The most commonly known dilemma is the identification problem, which 
means we can create different models that lead to the same outcome. Serial-stage and 
parallel-processing models of cognition try to explain human information processes in the 
form of decision trees or flow charts. Cognitive neuroscience models provide an attempt 
to describe human cognition as a set of neutral “hardware” and a set of “software” 
(Matthews et al., 2000). 

Due to the complex nature of human information processing, most research is 
done on a simpler subset of problems. Models of symbol processing are used to explain 
more complex processes. Most of these models describe serial processes, where only one 
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operation is performed at a time. Other models emphasize parallel information 
proeessing, where multiple operations take place concurrently (Rousselet, Farbre-Thorpe, 
& Thorpe, 2002). 

These parallel-processing models seem to describe many human cognition 
processes more accurately than serial models (Townsend, 1990). The spreading activation 
model is based on the explanation of categorizing stimuli. It is also closely related to 
association theories and deals with the relationship between long-term memory (LTM) 
and the way information is extracted from LTM to make decisions. Stimuli will tend to 
activate analogous concepts and this activation will spread to associate concepts. These 
concepts are also known as prototypes (Bagchi, Biswas, & Kawamura, 2000). 

Connectionist models are based on semantic networks with hierarchical structures 
and emphasize the learning of appropriate responses to stimuli. This semantic structure is 
described as a rule-based network which seems to be able to apply rules without being 
explicitly informed about the rules (Matthews et ah, 2000). The connectionist model 
seems to have the highest potential to explain human cognition. It explains a variety of 
human information processes without resorting to the biological level. However, it is 
based on symbol-processing concepts and may not be able to explain all human 
cognition. 

Other scientists, like philosopher John Searle, argue that computation has some 

inherent shortcomings, which means that it computation cannot capture the fundamentals 

of mental processes (Russel & Norvig, 2003). Those arguments are based on Godel’s 

“Incompletes” theorem which states that, when there are mathematical truths that can 

never be proven in a sufficiently strong mathematical system, any sufficiently strong 

system of axioms will also be incomplete. Searle (1992) argues that the attempt to equate 

thinking and information processing is an error, primarily because the interpretation of 

symbols, syntax, and semantics requires an outside intelligence to make the 

interpretation. Searle explains his own conception of consciousness as consisting of a 

"network" and a "background," but not in the unconscious/conscious framework of 

cognitive psychology. However, these opinions are mostly concerned with actual 

computational modeling of the human mind as it is applied in artificial intelligence 
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research (Russel & Norvig, 2003). In this study we need to understand the broad 
principles of information processing, attention, and resource management. Hence the 
models of human perception and cognition that have been discussed are valid for our 
purposes. 

2, Human Performance Models 

According to Matthews et al. (2000), “Performance may be viewed variously as a 
biologically-based activity supported by neural systems, or as a consequence of 
information processing “programs,” or as the outcome of an intentionally chosen 
strategy.” Pew (2000) sees human performance as the processes and effects of human 
behavior. 

Human performance models can be categorized within three major tracks: 

(1) Manual Control Models of human control in closed-loop systems 

(2) Task Network Models predict the probability of success and performance 
time. 

(3) Cognitive Architecture Models represent theories of human performance 
capacities and limitations. 

The manual control and task network models derived from an engineering 
perspective and are based on control theory. Control theory describes any system that has 
one or more feedback loops. These models range from simple linear models, which see 
human performance as a controlled engineering approximation that characterizes a 
nonlinear system as the sum of a linear transfer function combined with additive “noise” 
(Tustin, 1953), to more complex optimal control models (Kleinman, Baron, & Levison, 
1970), which accurately describe and predict problems such as human response times and 
many piloting tasks. 

Most combined models are called “hybrid models” because they merge various 
control-theory approaches with cognitive architecture models (Baron & Levison, 1977). 
Cognition and human performance are linked in cognitive architecture models. To 
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accomplish this linking, designers must address many factors and connections. Matthews 
et al. (2000) eah the major factors the “energeties of eognition,” whieh are assoeiated 
with coneepts of eapaeity, resourees, and attention. “Capaeity” is understood as the 
physiologieal and physieal limits of human performance. The term “resourees,” as used 
here, refers to the limitations of attention. A person may not have “enough” attention to 
follow multiple events that happen simultaneously. The eoneept of resourees is 
understood as eomprising the part of the human funetional eognitive arehitecture that 
supports eomputation. Resource theory is widely used to study “attention,” whieh is 
deseribed as having seleetive and intensive charaeteristies. 

The “seleetive” aspeet of attention refers to the mind’s tendeney to ehoose a 
eertain reaetion to a eertain speeihe stimulus, but no others (Matthews et ah, 2000). The 
term “divided attention” refers to the human ability to reaet to two or more things or to 
aeoomplish two or more tasks or mental aetivities at the same time (Wiekens et ah, 2004). 
“Sustained attention” refers to the mind’s ability to maintain its focus on one or more 
particular tasks or sets of stimuli over a longer period of time (Matthews et ah, 2000). 
Note that attention is not a singular eognitive proeess, nor is there a single, all- 
eneompassing dehnition that is universally aeeepted. Almost ah models of attention 
assume that there are normal limits to the amount of information that one ean proeess at a 
given moment in time. Cognitive resourees are limited, whieh ean affeet attention and 
other eognitive proeesses (Wiekens, 2002). 

Theories of seleetive attention have become inereasingly eomplex. Matthews et 
al. (2000) note, for example, that neuropsyehologieal and empirieal studies have found 
about 30 distinct modules with over 300 intereonneetions of the human brain involved in 
human vision. Attention models are divided into two major groups: stimulus-driven 
attention models and voluntary-eontrol attention models. Many arguments seem to 
support the idea that both reaetion to a stimulus and voluntary eontrol are involved in 
seleetive attention. Furthermore, the way the human brain ehooses to direet its attention 
may differ from ease to ease, depending on the environment and individual factors and 
circumstances (Peter, Sham, & Montague, 2000). 
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Divided attention is eonsidered to be even more complex. The main issue 
pertaining to divided attention, and thus the main point of attention resource theories, is 
cognitive resource management, which falls generally into two main categories: “single¬ 
pool” resource theories (Broadbent, 1958) and multiple-resource theories. Dual-task 
experiments have shown that “single-pool” theories are often not sufficient to explain the 
observed data. Multiple resource theories better explain human behavior in dual tasking. 

For this study, we found it important to assume that resources are hierarchically 
organized and that hierarchy seems to build functional blocks (Wickens et ah, 2004). For 
instance, spatial information might need the same resources independently from the 
particular physiological sensor. When we perceive visual spatial information, whether we 
hear, feel, or read it, however we perceive it, the information perceived seems to utilize 
the same cognitive attention resources. This phenomenon may explain why the addition 
of certain tasks may easily distract an operator from his main task, whereas other tasks 
have almost no influence on his performance and workload (VanRullen, Reddy, & Koch, 
2004). 

We know that humans are not able to focus on a single object for more than a few 
seconds, and that people have difficulty maintaining attention over a longer period. These 
ideas are supported by various studies of the 1940s and ’50s that did research on the 
performance of radar and sonar operators and their detection times and error rates 
(Matthews et ah, 2000). In this research, an operator’s ability to remain focused on a 
certain task is termed “vigilance.” Vigilance requires an operator to keep his attention 
fixed on one or more sources of information in order to fulfill a designated task. To 
illustrate “sustained attention,” researchers developed four major model theories: the 
filter theory, the expectancy theory, the arousal theory, and the resource theory (described 
by Matthews et ah, 2000). 

The filter theory predicts that vigilance tasks in which signals are present only for 

a short period will result in a quicker decrease in attention than tasks in which the signals 

are present for a longer period. The central idea here is that the brain’s mechanism for 

selecting information from an environment tends to be more attentive (i.e, to pay more 

attention) to novel information than to sustained information. Hence, over a period of 
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time, an operator is more likely to become distracted from the main task by outside 
information, that is, information unrelated to the task at hand. And, therefore, the 
possibility that he will miss important relevant information also increases over time. 

The expectancy theory states that operators/observers keep track of various bits of 
information. This enables the operator to extrapolate future information (i.e. predicting 
near future events). Over time, an observer’s mental estimate of the next probable event 
or information changes, based on his mental probabilistic model. When a task continues 
over a long period of time with only rare occurrences of specific or important signals or 
incoming information, the operator is likely to miss them because they are unexpected 
occurrences. 

Another theoretical approach that explains sustained attention is arousal theory. 
The basic idea of this theory is that sustained attention leads to a lowered level of arousal 
of the central nervous system (Matthews et ah, 2000). In implementing this research 
approach, researchers’ main tool is the analysis of data from recorded 
electroencephalograms (EEGs) used to monitor cortical arousal during vigilance tasks. 
However, some experiments show cortical arousal level decays that are almost 
independent from the character of the tasks. In some, the level declined even if no task 
was performed. 

The youngest theoretical model is the resource theory. Matthews et al. (2000) 
describe experiments that show that task workload plays a major role in vigilance decline. 
This theory suggests that, in demanding tasks, performance is more sensitive to fatigue- 
induced resource limitations than in simpler tasks. This position is supported by a number 
of single- and dual-task experiments (Wickens & Gosney, 2003). 

Eor our study, the expectancy theory and resource theory played a more important 
role than the others. Both theories were applied to derive both display design 
characteristics and interpretations of the experiment outcomes. 
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3. Skilled-Operator Performance 

In addition to assessments of spatial awareness, this study evaluates operators’ 
skills in the reeovery of an aireraft from unusual attitudes. We are eoneerned, therefore, 
in this regard only with models that explain skilled-operator performanee. Though human 
performance is also influenced by personality, age, lifestyle, stress, environmental 
factors, fatigue, and many other factors, the study is less concerned with those factors 
than with the operator’s skill level. Matthews et al. (2000) provide very helpful insights 
on these issues. 

Two sets of skills are usually involved in the accomplishment of a complex task: 
motor-skills and cognitive skills, each of which can be considered as a problem-solving 
component for specific types of tasks. Our task environment requires a combination of 
both. A pilot’s or operator’s ability to access the spatial orientation of his aircraft and to 
develop the control strategy necessary to alter its current spatial orientation is considered 
a cognitive skill. The actual steering of the aircraft, that is, the amount of input versus the 
reaction of the aircraft, is considered a motor skill (Headquarters Department of the 
Army, 1984). 

Psychologists usually classify learning by type, such as verbal, conceptual, 
perceptual, motor, problem-solving, or emotional. Other classifications refer to learning 
as intellectual skills, cognitive strategies, or attitudinal changes, along with descriptive 
terms like “surface” learning or “deep” learning. However useful these divisions may be, 
they are somewhat artificial. 

For example, a flight instructor may explain to a new student the procedure for 
entering a level left turn. This may include several steps: (1) visually clear the area, (2) 
add a slight amount of power to maintain airspeed, (3) apply aileron control pressure to 
the left, (4) add sufficient rudder pressure in the direction of the turn to avoid slipping 
and skidding, and (5) increase back pressure to maintain altitude. At this stage the student 
might demonstrate the turn using a handheld model of an aircraft. In this case, the student 
who can verbally repeat the instruction steps has learned the procedure by rote. He has 
acquired the cognitive skills necessary to then practice and develop the procedural steps 
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laid out by the instructor. None of this will be very useful to the student, however, if he 
never has an opportunity to make the level left turn in flight or if he has no knowledge of 
the function of airplane controls. 

With proper instruction on the use of flight controls and experience in controlling 
an airplane during straight-and-level flight, the student can consolidate both old and new 
perceptions into an insight on how to make a turn. At that point, the student will have 
developed the set of motor skills required to steer an aircraft in flight. Such 
understanding, though basic to effective learning, will still probably not be enough, 
however, to enable the student to make a correct turn on his first attempt. Only when he 
understands the procedure for entering a turn, has seen turns demonstrated, and has 
practiced turn entries himself until consistency is achieved, will the student have 
developed the skill necessary to apply what he has learned. At that point, both cognitive 
and motor skills combine in his performance of the stipulated task. In brief, the 
acquisition of skills can also be seen as the continuous elimination of error over time. 

4, Human Error 

This study addresses the nature of human error in the context of skilled aircraft 
operators and their interaction with nonhuman systems. Even experienced operators err 
from time to time. Thus reducing the error rate and the overall impact of errors is a major 
goal of human-performance research and of this study as well. Human error research 
links the concepts of situational awareness and human performance. The design of 
interfaces, in particular, is concerned with appropriate mental models and with SA within 
an environment made up of complex human/machine systems. 

The human/machine interaction in complex systems should be facilitated by 
appropriate interfaces that provide operators easy access to relevant information and 
support operators’ mental models/pictures of the system and the environment. Automated 
systems, in particular, tend to filter and hide information from the operator. Automation 
therefore may increase the “psychological” distance between operator and system 
(Wickens et al., 2004). To resolve that distance, researchers suggest improving the 

development of operators’ trust in automated systems the same way we develop our trust 
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in other people. If a system disappoints an operator, he may feel betrayed and his level of 
trust will deerease respectively (Dzindolet, Peterson, Pomranky, Pierce, & Beck, 2003). 

In 1999, Endsley noted that about 88 percent of the accidents among major 
airlines involve human error. Other researchers found similar numbers for air traffic 
control, automobile traffic, and military aviation (Matthews et ah, 2000). Matthews et al. 
define human error as an unintended or undesirable action, which arises from transaction 
between system requirements and operator action. To that definition we would add errors 
of omitted action (errors of omission). 

The taxonomy of human errors is broad. According to (Reason, 1990), human 
error and human error rates are more a reflection of people’s mental responses to tasks 
than correctable shortcomings. In this study, we focus on Rasmussen’s (1982) taxonomy 
of human errors, in which he suggests three levels of task performance based on the 
mental processing behavior employed at each level. He categorizes these as skill-based, 
rule-based, and knowledge-based domains, and from those domains, derives three levels 
of human error (Rasmussen, 1982). 

In Rasmussen’s skill-based domain, the tasks comprise very familiar actions 
performed in comfortable surroundings: the human is on autopilot. Rule-based tasks are 
also familiar to the performer. Upon his correct recognition of a situation or condition, 
the performer can apply a rule stored in his memory to steer him toward a known goal. 
Tasks in this domain tend to follow “if this, then that” logic. Knowledge-based tasks are 
those that are new, unfamiliar, or unique to the performer. Successful performance of a 
knowledge-based task depends heavily upon the performer’s fundamental knowledge, 
diagnosis, and analysis skills. Unlike his ability in rule-based tasks, in knowledge-based 
tasks the performer is unable to steer toward an end goal. Thus, knowledge-based tasks 
are best described as trial and error tasks. The performer’s behavior, not the task itself, 
dictates how and at what rate errors are made. What is rule-based to one performer will 
be knowledge-based to another. In other words, considering the human/machine interface 
is only half the battle. We must also consider the human/task interface. We will not 
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realize signifieant gains in human performanee until and unless we aehieve an 
appropriate performer-to-maehine mateh in a work environment in whieh the performer 
ean sueeeed. 

Reason’s (1990) model of human error, the Swiss Cheese Model, has been very 
influential in aeeident investigation beeause of its eomplexity and breadth. His 
framework is an expansion of a “resident pathogen” metaphor, whieh means that eausal 
faetors are present in a system before an aeeident sequenee oeeurs. That assumption leads 
to a differentiation between “aetive” failures and “latent” failures. Another important 
premise of the framework is that aeeidents eome from fallible deeisions made by 
designers and deeision makers. First, five basie elements of systems design are identified, 
whieh are then related to a given system’s breakdowns. One element is the deeision 
makers, those who set goals for the system and thus may make fallible deeisions, whieh 
Reason ealls “latent failures.” Another element is the line management, whieh 
implements the deeision makers’ (possibly faulty) strategies and whieh are also subjeet to 
defieieneies themselves, and thus are also ealled latent failures. A third element 
eomprises the preeonditions, eonditions that permit effieient and safe operations and, 
therefore, ean be preeursors for unsafe aets, that is, latent failures. A fourth element 
eomprises the produetive aetivities, the aetions performed by man and maehine that lead 
to unsafe aets and thus are ealled “aetive” failures. The final element eomprises defenses, 
the safeguards against foreseen hazards whieh may be inadequate and thereby result in 
aetive and/or latent failures (Reason, 1990). 

Reason divides unsafe aets into several different types. If the unsafe aetions are 
unintended, they are either “slips” or “lapses.” Slips are unintentional failures eaused by 
intrusions, omissions, reversals, disordering, or wrong timing. Lapses are memory 
failures that lead to the omission of planned items, plaee-losing, or forgetting. If aetions 
are intended, they are elassified as either “mistakes” or “violations.” Mistakes are either 
rule-based aetions, in whieh there is either a misapplieation of a good rule or an 
applieation of a bad rule, or they are knowledge-based aetions. Violations are routine, 
exeeptional, or deliberate aets sueh as sabotage. Some researehers argue that aeeidents 
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occur when a penetration of one of the various levels of the framework occurs. Latent 
errors combined with loeal triggering events may also lead to aecidents and incidents. 

Another important diseriminating issue is distinguishing a slip from other human 
mistakes. Slips often oecur in pereeptual-motor tasks and are considered skill-based 
errors. They are usually easy to observe and detect and do not represent either an error in 
a person’s understanding of a task or a lack of skills. However, in more complex 
environments, unlike with slips, it is hard to distinguish lapses from “real” errors, since 
the desired outeome and the appropriate operator’s aetion is sometimes not easy to define 
(Matthews et ah, 2000). Furthermore, the setup of an experiment should aceount for 
personality charaeteristics. Matthews (2000), for example, deseribes an investigation of 
substantial observable differenees in the personality profiles of fighter pilots who had had 
crashes and those who had not. Because of the scope of this study, we did not investigate 
aspeets of the partieipants’ personalities. 

A major critieism of researehers’ use Reason’s model of human error (1990) is 
that it may not be appropriate as an aceident causation model beeause of its linear 
sequence of levels. Reason does not eonsider intersecting influences that eome from 
various sourees. O’Hare (2000) has proposed a revised theoretical model and assoeiated 
elassification framework—called the “Wheel of Misfortune”—to help guide the accident 
investigation process. In O’Hare’s model, an individual operator’s actions are deseribed 
in terms of internal function taxonomy, a taxonomy that includes the skill/rule/knowledge 
framework that Rasmussen (1982) developed. A local “conditions circle” includes 
factors, such as the weather and the internal states of a fighter erew, that may be critical 
in an analytical breakdown of human performanee in complex systems. A global 
conditions circle considers the context within which the task activity, including the 
organizational proeesses, takes place. O’Hare’s model has three potentially valuable 
functions. Its idea of concentric spheres within spheres better represents aecident 
eausation than Reason’s linear sequenee of factors. Moreover, as an alternative to 
Reason’s “Swiss Cheese Model” (1990), the “Wheel of Misfortune Model” shows that 
the strength of a system is determined by its outer shell. O’Hare’s model is also good for 
direeting the attention of an investigator to specific questions within the layers of 
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concern, such as those regarding loeal aetions, the immediate realities of the operational 
environment, or the influenees of organizational funetioning. A final benefit, the Wheel 
of Misfortune model is expressed in terms of general proeesses that are not dependent on 
funetioning within any specifie domain, but allow information from other models to be 
used in its framework (O'Hare, 2000). 

Human errors do oeeur in highly familiar tasks for whieh operators are well 
trained and experieneed. These errors may be infiueneed by fatigue, personality, stress, 
and workload, and thus a system’s interfaees should eontribute to the reduetion of those 
faetors. If an operator’s training and experience are not suffieient to reeover from or 
overeome a eritieal situation, the system’s interfaees should provide highly intuitive 
information to help the operator understand the eurrent situation and find ways to solve 
the problem. 


5. Human Performance Measurement 

We defined “human performanee” as “a measurable outeome of a eertain task 
eonducted by humans.” Nevertheless, human performanee is notably eomplex in both its 
representation and its outcome. In many tasks, it is diffieult to describe the outeome 
parameters that are expeeted. For instanee, an attaek pilot has to aeeomplish a wide 
variety of tasks in order to fulfill his main task: to attaek a certain objeet. He has to 
operate the aireraft, navigate both relative to the ground and in spaee, eoordinate his 
aetions with wingmen, seareh and seleet the target, reaet to enemy fire, maintain radio 
eommunieation with several sourees, seleet the appropriate weapon, ehoose the right 
moment to fire the weapon, ete. Furthermore, eaeh of those tasks eonsists of various 
subtasks (Wiekens, 2002). 

Sinee sueh eireumstanees are often too eomplex for experimental scientifie 
researeh, we must eoneentrate on the elearly defined subtasks in a eontrolled 
environment. It is possible to assess the whole picture, however, by measuring all 
possible results: recorded flight parameters, radio eommunieations, and so on. However, 
that approaeh is outside the seope of this study, sinee we are concerned solely with the 

aspeets of spatial awareness pertaining to the design of flight displays. After 
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accomplishing the design, we have to test it within the context of eomplex flight tasks. 
Thus, we must apply specifie effectiveness eriteria to investigate how much the change in 
the independent variable contributes to the overall task performance and the change of 
the dependent variable. To isolate certain effects from uncontrolled factors, we must 
break down tasks into subtasks, as described. In addition, we have to simplify complex 
operations and minimize the influence of uncontrolled variables (Montgomery & 
Runger, 2003). 

Researchers use two major methods to measure human performance: objective 
measures and subjective measures. Objective measures are concerned with quantitative 
and qualitative outcomes of human performance. Therefore, we either measure physical 
variables like time and speed or we measure the quality of the outcome in terms of the 
error rate or the difference from the desired results. A subjective measurement assesses 
human performanee through questionnaires or self-reports; this is sometimes considered a 
weaker approach (Endsley, M.R. et ah, 2003). 

Objective performance-based measurements, also called measures of 
effectiveness (MOEs), or measures of performance (MOPs), are considered powerful 
tools for measuring the performance of a human-in-the-loop system and for identifying 
areas of inadequate situation awareness. The use of situations with testable responses can 
provide valuable insights into the user's situation awareness as well as how the user will 
act upon it (Endsley, M.R. et ah, 1998). 

Questionnaires facilitate insights into an operator’s cognitive model. By 
formulating the right questions, a researcher can gain insights as to why the operator 
acted one way and not another. Thus, using questionnaires helps the researcher explain 
the “hard” data of objective measurement methods. One weakness of this method is that 
questionnaires are subjective (Matthews et ah, 2000) and depend greatly on the way the 
experiment is organized and questions are asked. 

As stated previously, in this study we use a mix of performance-based 
measurement of spatial awareness and post-experiment questionnaires to evaluate 
subjective factors and better explain the objective measures (Prevot & Palmer, 2000). 
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C. DISPLAY DESIGN 

1. Implications of Human Performance and Cognition 

Display design principles should be consistent with human-cognition and 
performance models, which are based on human information processes and sensory 
systems characteristics. One possible perspective on display design is to think of how an 
entity’s location, orientation, and the world around it are conveyed, relative to three 
alternatives (Roscoe, Johnson, & Williges, 1980): 

Egocentric. The world and its symbology are presented from an inside-out 
perspective. The view through the windscreen of an aircraft or a forward-looking camera 
on a robot is egocentric. 

Exocentric. The world and the entity are presented from an outside-in 
perspective. Remotely operated vehicles such as slot cars or radio-controlled planes rely 
on exocentric views. 

Mixed perspective. The controller’s point of view moves with the entity but 
includes information about its orientation as well as the surrounding scene. Artificial- 
horizon displays with an icon depicting a plane’s orientation or tethered displays used in 
video games and virtual environments are examples of mixed views. 

Matthews et al. (2000) introduced three main types of displays: qualitative, 
quantitative, and representational. Quantitative displays are scales or counters. 
Qualitative displays are exemplified by warning lights. Representative displays provide a 
“working model” for the operator of the process being monitored. However, detailed 
principles and guidelines for display design are numerous. For instance. Clay (1993) 
addresses forty-six cognitive issues and in her final report lists over one hundred 
principles based on human cognition. We will focus on the most 
important issues and use Wickens et al.’s (2004) taxonomy of thirteen principles of good 
display design. Wickens et al. classify the design of displays in four major categories: (1) 
perceptual principles, (2) mental model principles, (3) principles based on attention, and 
(4) memory principles. 


25 



The perceptual principles comprise guidelines for good display design. The 
design should make a display legible. This is a basic prerequisite for increasing the 
usability of instruments. Avoid absolute judgment limits; the representation of a variable 
should be clearly assessable. An operator should not have to judge whether a certain 
threshold has been reached. This is particularly important for warning displays. Good 
displays support top-down processing, that is, displays are interpreted in a way people 
expect to perceive the information. For instance, in a bar display, the higher altitude is 
expected to be perceived as a longer vertical bar than the lower altitude. It would be 
counterintuitive if the altitude were represented as a horizontal bar graph. Another 
principle is redundancy gain. Important information is more likely to be interpreted 
correctly if the information is presented by more than one source. The last principle in 
this category is the rule that two information items should be presented in a 
distinguishable way: discriminability. If two information items are presented in a similar 
way, it is very likely that they are indeed similar in nature. This is often not the case in 
traditional flight instruments, which have similar designs but very different indicators. 
For instance, the altimeter and the instrument for airspeed both have a clock-like design 
(Figure 3). 

In sum, the mental model principles indicate that the presented information should 
be consistent with the operator’s mental model. The principle of pictorial realism means 
that a display should resemble the variable it represents. Wickens (2004) uses the 
example of temperature, which would be well represented in the form of a thermometer. 
The principle of the moving part requires that display elements move in keeping with the 
operator’s expectations and mental model (Roscoe et ah, 1980). 

A good illustration of this principle is the choice of either egocentric or exocentric 
flight instruments. Roscoe (2002) has evaluated fatal accidents due to horizon control 
reversals and so-called graveyard spirals. As part of his analyses of aircraft incidents that 
led to unexplained crashes to the ground, Roscoe conducted experiments to investigate 
the performance of pilots during instrument flights. He concluded that the cause of these 
kinds of accidents is best phrased as a question; What moves, the airplane or the world? 
His findings suggest that pilots may be confused by the way gyroscopic attitude 
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indicators represent the relative movement of the airplane. In particular, under high levels 
of stress or in situations where spatial awareness is almost or eompletely lost, even highly 
trained pilots may act inappropriately. They may interpret the attitude indicator as 
indicating that the aircraft is the moving part of the indieator and the horizon is fixed 
(Roseoe, 2002). Henee, the input given by the flight eontrol is reversed relative to the 
correet reaetion, the situation worsens, and the pilot is caught in a graveyard spiral. A 
control reversal usually occurs at night or in clouds, because in those situations the pilot 
has no external references to guide him. Roseoe (2002) suggests that flight instruments 
should support the basic mental model that the ground and horizon is fixed and the 
aireraft moves: the exoeentrie view is preferred over the egoeentric view. However, while 
all Warsaw-pact aircraft have exoeentrie flight instruments, and are therefore in keeping 
with Roseoe’s (2002) suggestions, most NATO aircraft have egoeentric instrument 
design (Cummings, 2003). Interestingly, other researehers found no evidence that one or 
the other of these views led to significantly different performances in flight tasks (Self, 
Breun, Feldt, Perry, & Ereoline, 2002). Previe and Ercoline (1999) present a 
neuropsyehological explanation for these findings. The main neuropsyehological 
argument is, aecording to the authors, that the human brain uses a peripersonal reference 
system when reaching for or manipulation objects near our body, a focal extrapersonal 
referenee system for visual search and recognition, and an action extrapersonal reference 
system for orienting to targets and navigation in topologieally defined space. Since the 
task of piloting an aireraft is a spatial navigation issue, the last principle of reference is 
extremely important as a foundation for our representation in space. Thus, it is not 
important whether an attitude format conforms the optical presentation of the outside 
world. It is important that the display format conforms to what has been termed the 
pilot’s psyehic reflection of that spatial situation. 

The attention principles form Wiekens et al.’s (2004) third set of guidelines. Good 
display design that tries to avoid the distribution of information over several loeations on 
an instrument panel requires minimizing information access costs. The proximity 
compatibility principle indicates that, if more than one source of information is neeessary 
to accomplish a task, the related displays should be in close proximity. This principle 
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facilitates information integration and supports divided attention. However, the prineiple 
applies not only to the spatial proximity of displays, but also to their shape or color 
proximity. The multiple resources prineiple guides different attention resourees with the 
same information and is important in situations where attention resources are limited or 
eompeting. Applying this prineiple makes it is possible for the pilot to proeess 
information aeross multiple resourees. 

The memory principles are based on the assumption that human memory, 
partieularly our working memory, is very limited. Therefore, the prineiple prescribes that 
memory should be replaeed by visual information and that we should not rely on the 
operators’ mental knowledge base (i.e., their knowledge in the world). For instanee, 
operators should not be foreed to memorize extensive proeedures. Those should be 
supported by eheeklists or other means. The predictive aiding prineiple is meant as a 
supplement to human weakness in predieting future events. In eomplex situations our 
working memory may be eompletely absorbed in assessing the immediate situation. 
Therefore, displays should inelude proaetive elements that support the human 
performance. 

The prineiple of consistency emphasizes that good display design should aeeount 
for other eommon display layouts that are familiar to the operator. For instanee, design of 
a new ear speedometer should adopt features of eommon speedometers sinee it can be 
expeeted drivers are already familiar with their design. 

In later ehapters we will see how Wiekens’ et al. and Roscoe’s prineiples are 
applied to the proposed instrument design. Some of the design prineiples have opposing 
goals and therefore they are diffieult to eombine. Moreover, some of the prineiples are 
less applieable in eertain domains than others. Nevertheless, all of them are based on 
researehers’ eurrent understanding of eognitive proeesses and, therefore, will likely 
support human performance in ehallenging aviation tasks. 

2, Implications of Situational and Spatial Awareness Models 

Endsley et al. (2003) analyzed the design proeess and design issues that support 

SA. They deseribe the methodology used to design a user interfaee to support SA (Figure 
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2). The first step is to analyze the SA requirements. This involves environmental 
eonditions and user eharaeteristies. The next step is to analyze the operational 
requirements and the applicable technology. This leads to a conceptual design, which is 
then assessed and optimized by a test and evaluation process. 



Figure 2. User interface design process 
(Endsley et ah, 2003, p. 46) 


Many of Endsley et al.’s basic implications are already reflected in display design 
principles implied by models of human performance and cognition. The authors 
emphasize aspects of attention tunneling, data overload, complexity, and incomplete 
mental models as major sources of poor SA design. To design a system for SA, Endsley 
et al. suggest conducting a goal-directed task analysis (GDTA). GDTA focuses on the 
basic goals of an operator and the major decisions the operator has to make. In addition, it 
generates a description of an ideal SA for the operator. 
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After conducting a goal-directed task analysis, Endsley et al. (2003) developed 
fifty-one principles for designing a system for SA. The principles partly overlap in their 
underlying mechanisms and are sometimes hard to discriminate. For this study, we will 
concentrate on a selection of those guidelines. 

Studies show that avatars may support spatial awareness because of their 
exocentric view (Lin, Abi-Rached, & Lahav, 2004). Lin et al. conducted experiments 
using the concept of virtual avatars, which are virtual objects that represent a participant 
or physical object in a virtual world. An avatar is a form of embodied person or entity 
outside the virtual world (Sherman & Craig, 2003). The avatar in Lin et al.’s (2004) 
study is embodied as an abstract airplane and was designed to lead the participant in a 
car-driver simulation through a virtual environment. Both motion prediction cues and an 
earth-fixed reference frame were claimed to reduce simulator sickness. The design helped 
to solve cue conflicts, led to a better understanding of the participant’s own movements, 
and increased user acceptance. For many reasons, such as the need to support an overall 
understanding of a current situation, to reduce the cost of scanning different flight 
instruments, and to lower the cost of cognitive integration, exocentric 3D-display designs 
are often considered superior for many situation awareness tasks (Tsang & Vidulich, 
2003). 

The first principle is to organize information around our goals. If we want to 
support spatial awareness, it is important to present information about spatial orientation, 
speed, and altitude close together, since they are the essential parameters needed to 
control the aircraft. How these information items are grouped is also important. Another 
principle requires that we present information directly and it implies that the information 
is intuitive. The operator should not be forced to calculate information. However, 
information filtering should be used carefully. The method used to abstract information 
should not lead to a situation in which essential information is oversimplified or hidden 
from the operator. A major goal of any SA design is to support global SA, a high-level 
general idea of situations across operator goals (Endsley, M.R. et ah, 2003). Hence, the 
display should give a comprehensive picture of the situation. For spatial awareness, this 
means that spatial orientation information is not enough. The aircraft’s position relative 
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to the ground and/or to other aireraft has to be displayed as well. In partieular, if we deal 
with complex situations, abstraction and reduction are necessary. The designer must 
decide which information has to be included in the display of main attention and which 
information can be integrated elsewhere. Another question is how much to simplify 
complex interactions and facts. The information should be consistent and avoid breaks in 
the logic. In regard to the choice of symbology and other pictorial metaphors, Endsley 
(2001) suggests using pictorial displays to support the user’s mental models. For 
example, in cases supporting spatial awareness, speed can be shown as a vector displayed 
as an arrow of varying length. Alternatively, the aircraft can be depicted by an aircraft 
icon instead of a simple line (Endsley, M.R., 2001). 

Concerning warning and alarm displays, Endsley et al. (2003) state principles that 
help an operator understand the nature, the importance, and the system-wide connection 
of a warning or alarm. They refer to it as “providing projection support.” A display 
should not show an isolated warning signal with no further information. Rather, it should 
show where in the process of the system the warning occurred and also show implications 
and possible information sources. Designs supporting spatial awareness should not only 
show an altitude warning, but should also integrate information about the current flight 
maneuver, the terrain, and the planned flight path (Parrish, R.V., 2003). An activated 
alarm should not distract the operator from his main task nor should it overlay main 
sources of information in such a way that it is no longer readable (Wickens et ah, 2004). 
The latter principle is particularly vital in aviation environments because an uninterrupted 
flow of spatial information is essential to conduct an appropriate recovery maneuver of 
an aircraft from an extreme flight situation. The most challenging design principle is the 
need to facilitate Eevel 3 SA by supporting the prediction of future events. This goal can 
be achieved by extrapolating current data or by “storytelling.” (Endsley, M.R. et ah, 
2003) “Storytelling” is a way to display data by integrating various variables into a time- 
based chain of events. These events should represent a logical combination of elements of 
a system or subsystem. An example of “storytelling” would be a display that showed an 
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aircraft model that represented the eurrent position and motion of the rudder in a pietorial 
way and also showed how the air-speed veetor ehanged aeeording to ehanges in the 
rudder position. 

Gush (2000) investigated the problem of how physieal systems, sueh as brains, 
represent themselves as subjeets in an objeetive world. If a human ean imagine spaee and 
ean eharaeterize himself as being somewhere in that spaee, then he will understand that 
he is one entity among others in a world that exists independently of him. Gush defines 
genuine spaee as “a multidimensional magnitude that has aetual extension, it is a region 
within whieh things, and even oneself, ean or eould physieally move abouf ’ (Gush, 2000, 
p. 54). He argues that humans prefer an egoeentrie representation of the world, in whieh 
the human is fixed in spaee and the world moves around him. Reeent researeh in aviation 
psyehology, however, suggests that pilots prefer an outside-in view of the world (Tsang 
& Vidulieh, 2003).These findings are supported by an earlier study, eondueted by Previe 
and Ereoline (1999). This study shows that the “Outside-In” attitude display coneept is 
superior in order to avoid roll-reversal errors during normal flights and in reeovering 
from unusual attitude. One early approaeh to utilize an exoeentrie instrument design was 
a study by Hennessy, Lintem and Collyer (1981), that used an exoeentrie view on a 
aireraft model to faeilitate the flight training of pilots. Hennessy et al. (1981) found that 
the outside-in display enabled the students to learn flight maneuvers faster than just using 
the traditional flight instruments. However, we were not able to traee any further 
development or applieation of this training eoneept. 

Display design prineiples based on analyses of the level of desired SA and spatial 
awareness show similarities to prineiples derived from human eognition and performanee 
researeh. However, SA-based prineiples are somewhat more goal-oriented and support 
deeision proeesses. 

3. Additional Display Design Considerations 

In addition to the display design guidelines deseribed above, there are other 
design issues that must be addressed. One important issue is the relative size and loeation 

of the display. Displays with important information or instruments that are frequently 
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used by the operator should be eentrally plaeed and distinguished from seeondary 
displays. Their size should provide optimal readability and aeeess. The same 
qualifieations apply to salienee. Salienee is defined as the eompellingness of eertain 
forms of information and is mainly defined by its physieal eharaeteristies (Endsley, M.R. 
et ah, 2003). The importanee of information and its salienee should eorrespond. 

Another design issue of displays and instruments is the avoidanee of elutter. 
Wiekens et al. (2004) suggest a number of methods for eontrolling elutter. In this eontext, 
we must eonsider the negative effeet of eolor elutter. Colors should be used sparingly and 
eonsistently. If a eolor is used to indieate that a limit has been reaehed or to represent a 
warning, that eolor should not be used again for other representations. The eommon 
“eultural” meaning of eolors should also be eonsidered. For instanee, the eolor green 
means something is “okay” or “proeeed”; red means something is “not right” or “stop” 
(Christ, 1975). 

Head-up displays (HUDs) and head-mounted displays (HMDs) in modem aireraft 
involve inherent issues for humans in regard to foeal distanees. When the human eye 
foeuses on the distanee in a HUD/HMD for a long period of time, it then has a problem 
refoeusing on greater distanees. This problem, ealled instrument myopia, is an improper 
aeeommodation of the eye during the use of an optieal instmment (Mouroulis, 1999). 

Another issue in display designs whether to use 2D or 3D representation. The 
answer seems to depend on the purpose of the display and the speeifie data to be 
displayed. Researeh suggests that task eharaeteristies demand task- speeifie display 
eharaeteristies. A 3D visual seene is a more natural, “eeologieal,” and eompatible 
representation of our 3D world than that provided by 2D eoplanar displays (Naikar, 
1998). Another benefit of a 3D seene is that a single integrated display reduees the need 
for mental integration of multiple 2D displays (Alexander & Wiekens, 2003). However, 
depth is represented less preeisely in 3D, beeause the display has to indieate distanee in a 
nonlinear way. And though a 3D display faeilitates the holistie pereeption of the seene, it 
is more ehallenging to make aeeurate estimates of relations or values along any axis 
(Mejdal et ah, 2001). Wiekens (2003) explains other disadvantages of 3D displays as the 

“2D-3D effeet.” Pilots subjeetively try to rotate veetors in depth to make them parallel to 
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the view plane. This leads to errors in orientation. However, 3D displays are useful for 
overall situation assessments, whereas 2D displays have advantages in supporting 
precise-control tasks (Parrish, R.V., 2003). 

Similar arguments apply to the designer’s choice between fixed viewpoints and 
dynamic viewpoints. This issue is especially of interest in 3D displays. Research supports 
the position that humans prefer different viewpoint characteristics for different tasks 
(Davenport, 1997). Often data fails to show the performance advantages of 3D displays 
(Mejdal et ah, 2001). 

Dynamic viewpoints in 3D displays have proven to be efficient. But in aviation 
environments, dynamic viewpoints might be risky, since controlling an aircraft often 
depends on fixed reference points (Tsang & Vidulich, 2003). However, for global SA, 
dynamic viewpoints do have useful applications (Wang, 2003). 

The following chapter will show how all these design principles, guidelines, and 
techniques are implemented in our proposed design approach. 
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III. PROPOSED DISPLAY DESIGN 


A, GENERAL BACKGROUND AND DESIGN 
1, Design Goals and Purpose 

The major goal of the proposed display design is to support the operator’s spatial 
awareness in an easily understandable and intuitive way. Traditional flight instrument 
design is based on historic design features that are abstract and nonintuitive (U.S. Army, 
1984). 



Figure 3. Traditional flight instrument design 
(Screenshot ^xomX-Plane™ flight simulation) 


To develop the skills necessary to interpret and use displays such as the common 
instrument design shown in Figure 3, a pilot must have extensive training. In extreme 
situations, fatal accidents have occurred because pilots misinterpreted their cockpit 
instruments (Roscoe, 2002). 

Since current display design has inherited many poor design features through its 
historical roots, we decided to introduce a new way to represent the major elements of 
flight dynamics. The purpose of our proposed design is to provide a unique, innovative 
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way to display spatial information in flight instruments. The display is not designed to 
substitute for any of the traditional instruments. Rather, it supports the operator of an 
aireraft in extreme or ambiguous situations by intuitively providing the appropriate level 
of spatial awareness. 

To aeeomplish these goals, we implemented as many of the design guidelines, 
prineiples, and ideas for human-eentered display design as was appropriate and pertinent. 
The main prineiple involved is to support the operator’s mental model of the spatial 
behavior of the aireraft (Roscoe, 2002). Our greatest ehallenge was to support the highest 
level of spatial awareness (Level 3 SA), the projeetion of an immediate situation into the 
future. The major objeetive of the experimental testing and evaluation deseribed in this 
thesis was to aehieve verifieation of the design eoneept. 

2. Basic Design Ideas 

Inspired by the idea of avatars in eommereial eomputer games (Bystrom, 1999) 
and by NASA’s work with pietorial displays for pathway-finding in landing proeedures 
(Parrish, R.V., 2003), the main design features, are an exoeentrie design eombined with 
pietorial metaphors of the eurrent spatial orientation. By animating the entire seene, we 
“tell a story” about what happens during a flight maneuver. In this eontext, we eould eall 
it a “miniature abstraet virtual world” that represents the aetual spatial situation of the 
aireraft. The avatar “lives” in this virtual world. 

In our display prototype, the aireraft is represented by a 3D wire-frame model that 
hovers inside an abstraeted box and replieates all the basie motions of a pilot’s real 
aireraft. The frame of referenee is a eoordinate system, eonsisting of X-, Y-, and Z-axes. 
Altitude is represented by a 3D bar along the Y- axis. The aireraft’s movement relative to 
the ground is displayed by moving objeets along the X-/Z- plane. 
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Figure 4. Screenshot: Overall display design 

The overall display layout facilitates the implementation into HUDs or HMDs. 
Principles of abstract and simplified symbology and restrictive uses of colors are applied 
(Figure 4). 


3, Color Coding 

The use of color is limited to red and green. In correspondence with color-coding 
design principles, green is used to show that the overall situation is not critical. Red is 
used for all warnings and critical conditions (Clay, 1993). 


4, Moving Elements 

The proposed display design has three major moving parts: (1) the model of the 
aircraft, (2) the ground objects, and (3) the altitude bar. All movements are designed to 
support an operator’s mental models. Every movement is intended to be natural and 
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meaningful (Endsley, M.R. et al., 2003). Information is integrated to eontribute to the 
eomplete situational pieture, and every moving part is designed to be on the same level of 
abstraction. 


5, Symbology and Pictorial Metaphors 

The symbology supports metaphors of real objects and conditions. The grade of 
abstraction depends on the level of abstraction in the real world. Therefore, the highest 
grade of abstraction is that of the spatial reference axis and the altitude. The lowest grade 
of abstraction is that of the model of the aircraft (Sherman & Craig, 2003). However, the 
symbology for a certain fact or variable is designed to reflect each one’s uniqueness so as 
to avoid misinterpretations. For instance, the ground is represented by squares, whereas 
the altitude is symbolized as half-transparent cubes. 

6, Relationships to Other Flight Instruments 

One important issue was the decision whether the display should be integrated 
into the layout of other flight instruments in the experiment. The 3D layout, the pictorial 
symbology, and the exocentric view are contradictory to traditional flight instrument 
design (Previc & Ercoline, 2004). However, the experiment is designed to determine 
whether a mix of exocentric and egocentric display designs lead to decreased human 
performance. 


B FRAME OF REFERENCE DESIGN 

1. Frame of Reference for Spatial Orientation 

To implement the exocentric design we created an abstract system of three 
perpendicular axes. This is a common way to represent the three spatial axes; therefore, it 
supports intuitive understanding of this abstraction. The two main axes are the X- and Y- 
axes. The X- axis builds the reference for the roll parameters; the Y- axis is a reference of 
the pitch angle. The Z- axis contributes to the depth impression of the scene (Figure 5). 
This system of axes builds an abstract box in which the aircraft moves. 
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We chose a visual angle of 55° as the field of view, which balances the need for a 
naturalistic look and the demand to make sufficient information available (Parrish, R.V. 
et ah, 1994). The viewpoint is fixed from outside the virtual box. 



Figure 5. Screenshot: Axes system without aircraft model 

2, Abstraction and Design of the Ground Representation 

There were two reasons we designed a animated ground representation: first, 
without a ground, the aircraft model would hover in space, which would result in an 
artificial, unrealistic look. Second, the ground is an important plane of reference for the 
movement of the aircraft model. On the other hand, details of the ground terrain are not 
important. Therefore, we modeled the ground by a number of square-shaped objects. The 
objects are generated at random positions and move against the main flight direction to 
maintain a high salience level. The squares are displayed as frames so they will attract 
little attention when the altitude is not critical. When the altitude becomes critical, the 
squares become solid. 
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The overall impression from the display is quite natural; it fits well into the 
general eomposition of elements. The eurrent design is limited to fiat terrain, but a 
subsequent implementation of height-mapped terrain is possible. 


C. AVATAR DESIGN 

1. Basic Design and Abstraction 

The airplane is implemented as an avatar. As stated previously, an avatar is a 
virtual objeet that represents a partieipant or a physieal objeet in a virtual world. The 
representation ean take any form. An avatar is in the form of an embodied person or 
entity outside the virtual world (Sherman & Craig, 2003). This avatar faeilitates the 
pilot’s identification with a virtual object (his own aircraft in our case). Virtual guidance 
avatars (VGA) are designed and used to escort operators through virtual environments 
(Lin et ah, 2004). We took advantage of this idea by designing the aircraft model as a 
VGA. The VGA was built as a 3D wire-frame model of an aircraft. This model shows 
just the main elements of an aircraft: the components are the fuselage, the wings, the 
rudder, and the horizontal stabilizers at the tail. 

The texture of the aircraft is designed to support the discriminability of all the 
possible spatial orientations. The topside has a green wire frame on a black background; 
the underside has a black wire frame on a green background (Figure 6). Furthermore, the 
topside texture of the aircraft’s body has only longitudinal lines, whereas the underside 
has squares. At certain view angles, the green lines of the upper part will be seen in a way 
that makes the topside look very similar to the underside (see Figure 8). To facilitate 
discriminability between the top- and underside of the aircraft model, two red rectangles 
were placed on the underside of the wings. 
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Figure 6. Screenshot: Top- and underside views of the avatar 


The aircraft is in level flight when the smallest possible silhouette is seen on the 
display (Figure 7). The 2D/3D effect described by Wickens et al. (2003) may cause the 
operator to want to adjust the model parallel to the ground, which would result in a 
positive (nose-up) pitch. When this display design is applied in real systems, that issue 
must be addressed in the training procedure. 



Figure 7. Screenshot: Leveled aircraft 
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2, Flight Dynamics Implementation 

The model of the aireraft moves according to the pitch and roll of the real aircraft. 
The display has no built-in flight dynamics: it relies solely on input parameters from the 
hosting flight simulation or actual aircraft. Neither the yaw nor the heading is displayed, 
since the operator expects the nose of the aircraft to be in front of him. Instead, we 
implemented the slip, which is a rotation around the Y- axis away from the main motion 
vector along the Z- axis. This movement occurs for example when crosswinds push the 
aircraft out of its orientation along the main heading direction. This implementation is 
consistent with the principles that support the operator's mental model of the spatial 
orientation and behavior of the aircraft. 

D, DESIGN OF ADDITIONAL INFORMATION IMPLEMENTATION 

1. Ground Speed and Movement Relative to the Ground 

The objects of the ground move along the flight direction to simulate the relative 
ground speed. The greater the speed, the faster the objects will move. The size of the 
objects changes with the altitude (minimum size 2mm x 2mm; maximum size 10mm x 
10mm). The higher the altitude, the smaller the objects become. This supports the mental 
model of the operator, because when one looks down from a height, details on the ground 
become smaller. The effect of shrinking object size induces the perception by the 
operator that, with higher altitude, the ground moves more slowly. When the aircraft 
starts a turn maneuver, the objects’ movement changes in keeping with the direction of 
the turn. They behave like ground details would behave in the same circumstances in the 
real world (Figure 8). 
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Figure 8. Screenshot; Turn maneuver (left turn) 


2, Altitude 

The altitude bar consists of a stack of semitransparent cubes. This design feature 
ensures that the Y- axis is still recognizable as the vertical reference. If the pile of cubes 
were red, the altitude would be interpreted as unsafe. Green cubes represent a secure 
altitude. Every 250 feet, a subsequent cube appears above the preceding one. The 
numerical value of the current altitude in feet is displayed parallel to the Y- axis. 

3, Warnings 

Moving ground objects, the representation of the X-, Y-, and Z- axes, and the 
body of the aircraft are presented in green when the situation is not critical. The 
longitudinal axis of the aircraft is displayed in green when the pitch angle is positive 
(upwards); it is coded in red when the pitch angle is negative (downwards). 
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If the situation is critical (an altitude below 2500 ft.), all elements of the scene 
turn red except the aircraft model. The red color-coding provides salient warning 
information and maintains the readability of the spatial information about the aircraft 
orientation. The ground objects have two stages during critical conditions. In the first 
stage, the frame of the objects turns red. In the second stage (for instance an altitude 
below 1500 ft.), the objects become solid red squares. This increases the salience of the 
objects (Figure 9). In combination with the growing size of the squares, this design 
provides an intense warning signal. 



Figure 9. Screenshot: Critical altitude 


E. PROGRAMMING AND IMPLEMENTATION 
1, Programming Environment 

To implement the proposed design idea, the graphics programming, network 
programming, and the complete project source code were developed using the 
programming language C++ and were compiled for Microsoft® Windows™ XP operating 
systems. The graphic application was developed in the OpenGL® version 1.4 and the 
OpenGL Utility Toolkit (GLUT®), a graphic library for C++. The program was 
developed and compiled using Microsoft® Visual Studio® version 7.1.3008. 
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The network sockets were based on the Public Library PLIB version 1.20 under 
the GNU Library General Public License, as published by the Free Software Foundation. 
All threads are based on the open-source library OpenThreads, version 2.1, under the 
GNU Library General Public License, as published by the Free Software Foundation. The 
program consists of two subprojects structured as shown in Figure 10. 


B l{^ Project 

[kal References 
B 01 Source Files 

^ AirPlane.cpp 
^ HUD_Box.cpp 
^ Particle.cpp 
^ ProjectMain.cpp 
^ SceneDisplay.cpp 
Q text.cpp 
Q UDPCiient.cpp 
B 0 Header Files 
[E] AirPlane.h 
[S] HUD_BOX.h 
[=] net.h 
[El netSocket.h 
[El Particle.h 
[El SceneDisplay.h 
[El text.h 
[E] UDPCiient.h 
[El ul.h 

f l Resource Files 
B ijp SimpleGLUT 

10] References 
B ^ Source Files 

Q SimpleGLUT.cpp 
B ^ Header Files 

[E] SimpleGLUT.h 
Resource Files 
[i] ReadMe.txt 


Figure 10. Software structure 


The most important class is the HUD Box-class. This class creates the spatial 
axes and the altitude indicator. It integrates the aircraft model and animates it according 
to the transferred data of pitch, roll, and slip. Furthermore, it is responsible for creating 
the flow of objects by instantiating the particle objects, randomly determining the start 
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coordinates, and providing data for the next movement. It also generates the numerieal 
messages and displays and evaluates data for generating warnings and the maneuver 
time. 

The aireraft model is generated by the AirPlane-class. The aireraft eonsists of a 
top- and underside polygon model. These polygon models are assembled so that the top- 
and the underside of the aireraft look different in shape and eolor and avoid transpareney 
(i.e., seeing through a wire-frame model). The partiele elass generates a single partiele as 
a square and eomputes its next position based on the given speed and movement angle. 
The SceneDisplay-class instantiates the HUD-Box by determining its size, positions the 
box in the window, assures appropriate viewpoints and view angles, generates pop-up 
menus, and supports keyboard inputs. 

The text and the UDPClient-classes are helper classes that generate text displays 
in OpenGL and provide the UDP soeket eonneetions and the data transfer from the flight 
simulation into this program. 

The simpleGLUT-class provides the OpenGL environment to ereate a window in 
the operating system of the eomputer. It works together with the ProjectMain-class, 
whieh is the program’s entry point. It starts the whole program and determines eertain 
initial values. 

2. Data Representation and Network Communication 

The applieation reeeives data from any flight simulator via network 
communieation using the UDP standard. The typieal UDP port is 49001. Other 
eonfigurations are possible. The UDP elient demands the X-Plane™ data strueture 
speeifieations. The variables being reeeived are angular values for pitch, roll, and slip in 
degrees. In addition, the applieation needs information about the speed over ground, the 
altitude, and the elapsed simulation time in seeonds and the turn rate in degrees per 
second. 
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IV. METHOD 


A, METHODOLOGY 

1. Experiment Goals and Purpose 

The goals of this study are to design, develop, and implement a display design 
that supports or enhanees spatial awareness in 3D orientation tasks. To test and evaluate 
whether these goals are accomplished we must assess the impact of the proposed display 
design on people’s spatial awareness in an experimental environment. 

The overall goal of this experiment is to evaluate the design concept, not to 
answer all the questions about design variations or explore all design features. Our 
purpose was to achieve an overall assessment of the design idea and the underlying 
principles in a hierarchically-structured experimental environment. 

2, Research Questions 

The study is limited in its scope and assets. Our evaluation sought to answer four 
main questions: (1) How will the display design, as compared to a traditional display 
layout, influence the time it takes to judge a static spatial orientation? (2) How will this 
display design, as compared to traditional instrumentation, influence a complex 3D 
orientation task? (3) What is the operator’s subjective impression of the design? (4) Will 
the design’s impact on participants with a strong aviation background be different then 
that of participants with no aviation background? 

Answers to the first question provide insights into the ways people build mental 
pictures of their current spatial situation. The time it takes them to mentally construct 
spatial awareness enables us to judge the effectiveness of the proposed design’s support 
of the human mental process that generates spatial awareness. 

Answers to the second question reveal whether the display can influence a 
complex task like a recovery maneuver. This question marks the experiment’s transition 
from an isolated view to a goal-based task, which involves factors such as motor skills. 
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cognitive skills, and procedural knowledge. It allows us to estimate the impact of the 
display on real-world tasks and enables the participants to evaluate the design in the 
context of a goal-based task. 

Answers to the third question give us insights about the impression of the design 
on participants beyond purely objective performance measurements. These insights 
enable us to bring the MOEs into the context and may help in the diagnosis of ambiguous 
outcomes. 

Question four provides information about the relationship between an operator’s 
experience level and/or training and his acceptance of the proposed design. A pilot with 
hundreds or thousands of hours experience with traditional flight instruments will 
perhaps find it easier to continue to interpret and use the old instrumentation and harder 
to adapt to the new one. The opposite might be the case for nonaviators, because, to those 
participants, both design representations will be novelties. 

3. Constraints and Assumptions 

Since the experiment was to involve human participants, the experimental plan 
had to be submitted to the Institutional Review Board (IRB), which approved the 
experiment as proposed (Appendix A). All participants were pre- and post-screened for 
any symptom of simulator sickness. 

The experiment inherited the technical limitations of the flight simulator that was 
used, a low-cost, fixed-base, 230° field-of-view helicopter simulator. The flight dynamics 
were provided by a commercial flight simulation program, X-Plane®, by Laminar 
Research™. In version 7.13., X-Plane® does not allow storage of complete flight 
situations, including flight dynamics. As we will describe later, that disallowance 
influences the degree to which we can replicate identical situations. Another limitation is 
that the flight simulator only accepts the current position and the flight dynamics of the 
aircraft from X-Plane, not the spatial orientation, or the spatial attitude of the aircraft. The 
simulation program then generates its own environment based on that data. This 
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limitation makes it impossible to create clouds and other environmental effects. Visibility 
is thus determined by the amount of daylight and the level of haze. 

The cockpit of the flight simulator was equipped with genuine flight instruments. 
However, since the simulator was a helicopter simulator, the spatial attitude indicator was 
limited to ±90° for roll. This limitation made using those instruments undesirable. We 
decided to use the original cockpit layout generated by X-Plane. To accomplish that, we 
used black fabric to hide the flight simulator’s original instrument panel and placed a 15” 
monitor with the visual cockpit layout of X-Plane in front of it (Figure 11). 



Figure 11. Flight simulator instrument panel 


We also had to change the model of the flight simulator’s flight dynamics, since 
the flight dynamics of a helicopter were not appropriate for our experiment’s extreme 
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flight situations. We chose the model flight dynamics of a Piper PA-46 Malibu, a small 
single-propeller turboprop aircraft that seats five passengers. The model was chosen 
because we expected that this might be an easy way to control the airplane model. 

The input devices were the helicopter’s control devices, the stick and pedals. The 
throttle was set to a fixed value of 80 percent engine power. The operator was not able to 
change the power or to trim the aircraft, and the simulated aircraft usually had a tendency 
to increase in pitch with released control devices. 

The flight simulator makes it possible to have acoustic in-flight sound. This 
feature was switched off during all experiments, however, because the given situation 
was that the participants were flying a UAV, rather than sitting in a real aircraft. 

We decided that, if the aircraft crashed (which happened in two cases), the trials 
would not count. Our assumption was that a crash does not reflect the normal ability to 
recover the aircraft and was therefore an “exceptional” event. Another compromise we 
made was to accept that every provided flight situation in the recovery experiment was 
completely random. A participant would freeze the simulation in all equally difficult 
attitudes. Those situations would not be exactly equal in degrees of pitch and roll, 
however, nor in their flight dynamics or wind conditions. We maintained the level of 
difficulty and the number of trials: six. 

4, Technical Equipment 

The experiment took place at the Naval Postgraduate School, Monterey, 
California, Watkins Building, room 212B. The flight simulator uses a three-sided 
projection cubicle, the Cave, consisting of three large panes (6 ft. x 8 ft.). The computer¬ 
generated images were projected on the panes by back projection with a 1024 x 768 pixel 
resolution. Each pane uses one computer and one projector and provides a frame rate of 
about 65 frames per second (Figure 12). The participant’s position was determined by a 
distance to the front panel of 1.55 meters and 0.97 meters to each side panel. 
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Figure 12. 


HOST FLIGHT MODEL 

NFS flight simulator equipment 


The partieipant was plaeed on a platform three feet high in the eenter of the Cave. 
This ensures 230° field-of-view (Figure 13). 



Figure 13. NFS flight simulator 
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The proposed display was implemented on a 10.4-inch TFT monitor, which was 
mounted at the top of the instrument panel (Figure 14). The size of the actual scene 
displayed was 80 mm x 80 mm. The participant’s viewing distance was 640 mm and the 
display was off-center 24° to the left. The average eye-height of the participants was 960 
mm. The display monitor was mounted at a height of 1,050 mm. 



Figure 14. Flight simulator cockpit 

The time of the first sub-experiment was stopped by hand using a digital hand 
stopwatch, type SPORTLfNE®. 

5, Data Collection Methodology 

The time of the first sub-experiment was stopped to allow assessment of the 
current spatial situation by hand, with an estimated precision of ±0.1 seconds. The 
precision of the spatial judgments was measured by evaluating the drawings for pitch and 
roll from the provided schema. The drawings were evaluated in 15° steps based on their 
error from the given values. Hence, the precision was ±15°. Errors within 15° were 
rounded to the next value. Eor instance, an error angle of 17° would be counted as a 15° 
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error, a 26° error would be eounted as a 30° error, and so on. The data of the seeond sub¬ 
task was measured by a built-in funetion of the used display software. The preeision was 
±0.1 seeonds. The timing started when the partieipant began his/her experimental task. 
The initial flight situations were provided with a preeision of 1° for every axis. 

The use of subjeetive self-assessment provided a seale of five judgments in every 
questionnaire. Demographie data was eolleeted by a pre-experiment questionnaire 
(Appendix D). 

All the partieipants were exposed to all the instrument setups and all the sub¬ 
experiments. The order of presentation of the two displays was pseudo-randomized. Odd- 
numbered partieipants began with a traditional instrument setup; even-numbered 
partieipants began with the new instrument setup. This proeedure ensured that learning 
effeets were eounterbalaneed over the experiment. 

6, Data Analysis Methodology 

The data was reeorded using spreadsheets. Every sub-experiment allowed the 
partieipant six individually measured and reeorded trials. The data was analyzed by 
eommereial statistieal software using various methods, for example, linear regression 
analysis, residual analysis, model-fitting, and paired t-tests. The statistieal analysis 
foeused on the differenees between the measurements. To analyze the subjeetive 
statements, absolute values were used. 

Two major questionnaires were administered in the experiment. One 
questionnaire was subdivided into a pre-experiment simulator-siekness questionnaire and 
a pre-experiment demographie questionnaire. The other questionnaire was subdivided 
into two post-experiment subjeetive-evaluation questionnaires and one post-experiment 
simulator-siekness questionnaire (Appendix D). The questionnaires were administered 
for three reasons: to develop a seeondary method for determining the effeets of the 
dependant variables; to sereen partieipants’ symptoms of simulator siekness; and to 
analyze the relationship between a partieipant’s subjeetive and quantitative data. 
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B, EXPERIMENTAL DESIGN 

1, Basic Experimental Design Ideas 

The experiment was designed to eompare differenees in speed and precision 
changing only one variable (display design). The experimental design was a randomized 
“within subjects” design. The two groups of participants were compared in a “between 
subjects” manner. The assignment to the groups of pilots and non-pilots was based on the 
subjects’ flight experience and, therefore, not randomized. (Montgomery & Runger, 
2003). 

The dependent variables were the outcomes in terms of differences in speed, time, 
and precision. The independent variable was the instrument layout and design with two 
levels, old and new. 

Two sub-experiments were developed. The first was a static task, to measure the 
time and accuracy of the participants’ assessment of their spatial situation. It also served 
to promote familiarity with the use and interpretation of the instruments and displays. The 
second sub-experiment was designed to evaluate the time a participant needs to recover 
an aircraft from an unusual attitude while using the two different instrumentation setups. 
It assumed that a participant uses the knowledge gained from the previous experiment to 
accomplish this task. Therefore, the order of these sub-experiments was not random. 

The participants were told that they were the operator of a combined rotor-/fixed- 
wing-based unmanned aerial vehicle (UAV), type Eagle Eye, that the UAV is on a test 
flight and enters a sector of bad weather, and that the data-link connection to the UAV 
was interrupted because of weather conditions (simulated by blindfolding). When the 
data-link connection is reestablished, the task is to recover and to stabilize the aircraft at a 
safe altitude (Appendix G). 

2. Overall Experimental Design 

Every participant went through a series of experiment blocks of trials, shown in 
Eigure 15. Eirst the participant was introduced to the experiment. Then he filled out the 
necessary paperwork, including a consent form, a privacy statement, the pre-experimental 
questionnaire and a minimal-risk information form (Appendix C). 
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This part took approximately 15 minutes. After the initial briefing, the partieipant 
had the opportunity to beeome familiar with the flight simulator and eontrol devices in a 
free, self-paced flight training assisted by the experimenter. 

The pre-experiment training took approximately 10 to 15 minutes. Then, the 
participant started with task one, the given instrumentation setup, that is, with or without 
the new instrument. Next, the participant started the second subtask, the recovery 
assignment. Each experiment block was scheduled for 30 minutes. 

After accomplishing the first block of experiments, the participant was allowed a 
10-minute break. Then the participant started the second experimental block. When 
finished, he/she was asked to fill out the post-experimental questionnaire. This completed 
the experimental session. The detailed experiment is given in the experiment protocols 
(Appendix B). After a participant completed an experiment session, he/she was asked to 
stay for another 20 minutes to avoid any aftereffects due to simulator sickness. 


55 




Figure 15. Experiment schedule 

3, Static Spatial Awareness Sub-experiment 

The detailed procedure of the static spatial awareness sub-experiment is 
documented in the experiment protocol (Appendix B). The participant was led into the 
flight simulator, shown all the equipment, and instructed to sit in the operator/pilot 
compartment. The experiment leader explained all flight the instruments and controls, 
including the “WEBER-Box,” the term we used for the proposed display during the 
experiment. 

The participant was allowed to “fly” for about 10 minutes to become familiar with 
controlling the aircraft. After the training flight, the experimenter briefed the participant 
for the first task, recognition of the aircraft’s spatial orientation (Appendix G). Then, the 
instrumentation monitor and the WEBER-Box were switched off, in whatever order was 
applicable for that participant. 

The first trial was a practice trial. The participant blindfolded him/herself by 
wearing opaque goggles. The experimenter chose a pair of pitch and roll angles from a 
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set of possible angle pairs and transferred them via network to the flight simulation 
program. The leader then told the participant to start, and the participant took off the 
goggles and began to look at the instruments. He was instructed to say “Stop” when he 
had achieved sufficient knowledge of the aircraft’s spatial orientation to draw the pitch- 
and-roll on the evaluation sheet. When the participant said “Stop!” the experimenter 
switched off the monitor and measured the elapsed time. Then the participant drew 
his/her opinion about pitch-and-roll into the provided schema on the evaluation sheet 
(Appendix E). The experiment resumed with the next experiment loop and a new set of 
pitch-and-roll angles until the participant finished six tasks, plus one test trial. 

4, Dynamic Recovery Experiment 

In this experiment, the participant was advised to first read the task instructions 
(Appendix G). After starting the simulation, the experimenter provided flight commands 
to establish the aircraft in an unusual attitude (pitch-and/or-roll > 90°) above the secure 
altitude (> 4,000 ft.). Due to the technical limitations, the given flight tasks were slightly 
different from one another. The experimenter had to make sure that the level of difficulty 
for each flight situation was consistent with the others. When the aircraft was in a desired 
attitude, he froze the simulation. During this phase he continued to provide flight 
commands, to cover-up the artificial pause and to avoid letting the participant know that 
the simulation was actually paused. Then he gave the instruction “Start recovery” and the 
participant removed his goggles. The participant recovered and leveled the aircraft (±10° 
in pitch-and-roll) for at least five seconds, and the leader noted the maneuver time on the 
evaluation sheet for later evaluation (Appendix E). 

A crash of the aircraft was considered an invalid trial. If the participant crashed 
the aircraft during his practice trial, he was given another test trial. The experiment leader 
would start a series of new experiment trials until the participant had completed six valid 
experiment trials in each condition. 
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C. PILOT STUDY 

The pilot study was conducted with three volunteer participants to test the 
procedures, apparatus, and tasks. Another goal was to predetermine the number of 
participants that would be needed to achieve a statistical power of at least 0.80. 

Data from the pilot study showed that participants often dropped rapidly below 
the secure altitude (2,500 ft) during the recovery task. This introduced unwanted side 
effects into the experiment, since in those cases the time to re-regain a secure altitude 
added to the recovery time. This was not a desired effect. To fix it, we modified the 
experiment procedure and started the recovery task at an altitude of 4,000 feet. 

The pilot study was based on an early goal of the experiment: to get information 
about whether the new instrument design influences additional workload. For this reason, 
we developed two additional tasks: a mental rotation task and an acoustic spatial task. 
The idea was based on Wickens’s (2004) resource theory. We wanted to overload the 
attention resources responsible for solving spatial tasks. Mental rotation tasks are a very 
practical way to induce different levels of mental workload in a controlled way (Shepard 
& Metzler, 1971). The newly-developed mental rotation task was tested in an academic 
environment to determine the appropriate level of difficulty. The participant sees two 
similar-looking 3D shapes, which are rotated relative to one another. The participant must 
determine if the shapes are equivalent or different. This test caused an unexpected 
outcome, however, during the preliminary investigation. The participants had to look at a 
second monitor while recovering the aircraft and raise their left arm if they perceived that 
two shapes were not equal. Not a single participant paid attention to the secondary task, a 
mental rotation task, during the first seconds of the aircraft recovery. Furthermore, the 
recovery maneuver usually took them about 10 seconds. The rate that different shapes 
were shown was 7 seconds. Hence, there was just a chance to see two, maximally three, 
different mental rotation tasks during a recovery maneuver, but this was statistically 
almost impossible to evaluate. Therefore, we considered a different way to implement 
additional mental tasks. 

Using the same idea of overloading mental resources, we developed a spatial 

acoustic randomized task. In this test, the participant hears the spoken word “left” or 

58 



“right” on his left or right side. Taking advantage of the two stereo channels, we designed 
the test so that the words sometimes occurred on the correct side, sometimes on the 
wrong side (i.e., “left” would be heard from the participant’s right side). The participant 
again had to raise his arm if he detected an error. This method of inducing spatial 
workload turned out to be very effective. In addition, the rate of information was much 
higher than the application of the mental rotation task, and it was also possible to start the 
secondary task during the time the participant was blindfolded. The results were similar 
to those of the graphical task. No participant paid attention to this secondary task during 
the intense phase of the first seconds of aircraft recovery. 

We interpret this outcome as support for selective attention theory, since the main 
task was such a high priority for the participants. As a result, we decided to conduct the 
actual experiment without any secondary task, leaving the investigation of the influence 
of the proposed design on additional workload to future research. 

D, PARTICIPANTS 

The participants were randomly assigned on a volunteer basis from students and 
faculty of the NPS. The study involved 20 individuals (19 male and 1 female). Ten 
participants had a strong aviation background: there were 3 helicopter pilots, 6 fighter-jet 
pilots/naval flight officers, and 1 flight student. The mean age of the pilots group was 35 
in a range of 27 to 40. The average flight experience was 1200 flight hours in a range of 
30h to 3200h. The mean age of the non-pilots was 32 with a range of 21 to 60. 

All participants were pre- and post-screened for any symptom of simulator 
sickness. No participant reported any symptoms of simulator sickness. 
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V. DATA ANALYSIS 


A. HYPOTHESES 

1. Primary Hypotheses 

One primary hypothesis was that a person, using the proposed instrument design, 
would more quickly assess a spatial situation accurately than if he were using traditional 
instruments. This hypothesis can be explored via the objective results of the static spatial 
awareness experiment outlined here. The time a participant requires to assess a particular 
spatial situation indicates his ability to create a mental model of the current spatial 
situation. The accuracy of his mental model is measured by the angular errors, for pitch- 
and-roll, from the given spatial orientation. 

The time difference that results from a participant’s use of the two display 
designs, the proposed and the traditional, indicates that individuals will assess the 
situation as fast, or faster, with the proposed display: 

A time < 0; with A time = time(proposed design) - time (traditional design) 

This claim should be supported by outcome of the post-experiment 
questionnaires. We determined a priori that the minimum practical relevant time 
difference is 2.0 seconds. If we compare this to a real-world situation, with a given 
airspeed of 200 mph, the distance traveled in 2.0 seconds would be approximately 180 
meters. If an actual unusual attitude situation includes a rapid decent, this distance could 
be of critical importance. 

Our second primary hypothesis was that a person using the proposed instrument 
displays is at least as fast in recovering an aircraft as one using traditional displays. This 
hypothesis can be explored via the objective results of the recovery experiment. Equal 
recovery times would indicate that the new design is at least as efficient as the traditional 
design. Since in the recovery task the new instrument was added to the common 
instrumentation, equal recovery times would also indicate that the new instrumentation 
did not interfere with the recovery or confuse the participant. We link the recovery time 
to spatial awareness by assuming that the operator’s spatial awareness is what enables 
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him to decide which sequence of maneuvers he to will use to accomplish the task. We 
assume also that with a greater spatial awareness, the operator would explore the flight 
maneuver and/or the control-feedback loop more efficiently. This assumption should be 
supported by the post-experiment questionnaire. Since a recovery maneuver is a complex 
task drawing not only on spatial awareness but also on skills, knowledge, procedures, and 
decisions, our hypothesis is that the recovery time using the proposed display will be at 
least equal to the recovery time using traditional instruments. The threshold for this claim 
is again a limit of 2.0 seconds, based on the previously explained reasoning. 

2. Secondary Hypothesis 

As we noted above, the measured angular errors, for pitch and roll, from the given 
(true) spatial orientation indicate the accuracy of the participant’s mental model. Thus, 
our secondary null hypothesis was that the angular precision of the assessed spatial 
situation would be equal for the two instrument designs, the new and the old. 

A negative sum of angular errors would indicate a higher accuracy for the 
proposed design in the spatial awareness assessment. Equal accuracy would indicate that 
the new design was at least as efficient as traditional designs. We can assume that 
traditional flight instruments successfully evaluate the spatial orientation of an operator’s 
aircraft. Thus, the proposed display design should at least meet that threshold. 

The precision of spatial judgment in terms of angular errors was measured in 15° 
steps, that is, errors within 15° were rounded to the next value. For instance, an error 
angle of 17° would be counted as a 15° error, a 26° error would be counted as a 30° error, 
and so on. 


3, Baseline Determination 

First, we were interested in determining the number of participants needed to 
achieve a statistically significant outcome. We used statistical power analysis, in which 
power is the probability of rejecting a false statistical null hypothesis (Montgomery & 
Runger, 2003). Properly designed experiments must ensure that the power will be 
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realistically high, so as to detect reasonable departures from the null hypothesis. 
Otherwise, an experiment may not be worth doing. The estimate of the number of 
participants needed was based on the results of the pilot study. 

The level of significance was determined by a = 0.05, the desired statistical power 
0.70 to 0.85, and the minimum of practical relevant differences in time were 1.5 to 2.5 
seconds. The standard deviation was estimated to be 1.3 seconds. The results shown in 
Table 1 were calculated by MINITAB® Release 4.12.0. 


Table 1. Number of participants for different levels of statistical power 


Difference in Seconds 

Statistical Power p 

Number of Subjects 


0.75 

12 

1.5 

0.80 

13 


0.85 

15 


0.75 

7 

2.0 

0.80 

8 


0.85 

9 


0.75 

5 

2.5 

0.80 

6 


0.85 

6 


We designed the experiment to detect a minimum difference of 2.0 seconds and a 
statistical power of 0.8 at a level of significance of a = 0.05. Hence, at least eight 
participants were needed per group. If we were able to gain a better P-value than 0.05, we 
determined the level of significance would be a = 0.05. 


B, GENERAL DATA ANALYSIS 

1. General Statistical Description of the Data 

During the experiments, 1,216 data points in different categories were collected. 
The data analysis was supported by statistical software packages MINITAB® Release 
14.12.0 and IMP® version 5.1.2 . 
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The data categories were: 


(1) Orientation Time: The time in seconds that a participant needed to 
develop his/her spatial awareness. 

(2) Pitch Error: The angular error in degrees of the estimated pitch in 
degrees. 

(3) Roll Error: The angular error in degrees of the estimated roll of the 
aircraft. 

(4) Recovery Time: The time in seconds that a participant needed to 
recover the aircraft from an unusual attitude. 

(5) Demographic questionnaire: Age in years and flight experience in 
hours. 

(6) Post-experiment Questionnaire: The subjective assessment of the 
experiment. 

2, Demographic Data 

As previously noted, the nonpilot participants and the participants with aviation 
background had a similar age means. The non-pilots had a much higher standard 
deviation, mainly due to the one 60-year-old participant (Table 2). The detailed statistics 
are shown in Eigure 16. The one female participant was a member of the pilot group. The 
number for N = 240 results, from the fact that we had 20 participants with 2x6 trials 
each, thus a total of 12 trials per participant. Since in each of the twelve trials the 
demographic data is equivalent, it does not result in statistical flaws. No participant 
reported any symptom of simulator sickness. 


Table 2. Statistics of participants’ age 



Mean 

StDev 

Minimum 

Maximum 

All 

33.700 

7.672 

21 

60 

Pilots 

35.200 

3.502 

27 

40 

Non-Pilots 

32.200 

10.070 

21 

60 
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Figure 16. Statistics of participants’ age 


Using the flight hour statistics, we concluded that the group of pilots could be 
considered as very experienced. Most of the pilot participants had more than 1,000 flight 
hours. Only 3 of the 10 pilots had less than 1,000 flight hours. The details of the pilots’ 
flight experience are shown in Figure 17. All the pilots were trained in recovery 
maneuvers and all had experienced at least one spatial disorientation during their aviation 
career. 
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Figure 17. Statistics of participants’ flight experience 


3, Orientation Time 

According to an Anderson-Darling normality test, the orientation times cannot be 
considered as normally distributed (Figure 18). Since we were only interested in the 
individual differences, this does not concern us at this phase of the analysis. 
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Figure 18. Orientation-time statistics 

The statistical evaluation of the orientation times shows that the standard 
deviation for pilots, 2.8 seconds, was much smaller than for non-pilots, 3.7 seconds. This 
could be explained by the fact that, for unknown reasons, one of the nonpilot participants 
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an unusually long time; 25 seconds. Furthermore, the selective process of potential 
military pilots filters out persons with weak spatial-orientation abilities. 

However, the mean of both groups differs only by 0.07 seconds. Hence, they were 
practically identical. This result indicate that the underlying mental processes used to 
assess a spatial task were equal for the two groups and were not changed by training or 
experience. 

4, Angular Pitch Errors 

Figure 19 shows the overall distribution and statistics of the angular pitch errors. 
The statistics of the pitch error show that the non-pilots were able to judge the current 
pitch angle of the airplane as accurately as their pilot counterparts were. The variance 
was similar for the two groups (Figure 19). 
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Figure 19. Angular-pitch error statistics 


According to an Anderson-Darling normality test, we cannot assume a normal 
distribution of the angular pitch error. However, the box-plots show some outliers, which 
we decided to keep in the data set. In our opinion, the outliers represent human errors that 
may occur and, therefore, are not an abnormality. Angular errors, in particular, can lead 
to serious aviation mishaps (Previc & Ercoline, 2004), this study was also concerned with 
the possible reduction of that type of error. 
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5, Angular Roll Errors 


Roll Error 

Group: Pilots 
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Figure 20. Angular-roll error statistics 


Figure 20 shows the overall distribution and statistics of angular roll errors. 
Accuracy and variance of the judgment of their roll angle were almost equal for both 
pilots and non-pilots (Figure 20). Pilots were slightly better at estimating the roll of their 
aircraft, but the difference of 15° for a maximum error of 180° was considered negligible. 

According to an Anderson-Darling normality test, the distribution of the angular 
roll error cannot be assumed to be normally distributed. The statistics show some outliers, 
which we decided to keep in the data set. 
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6, Recovery Time 

The statistics on the recovery time needed revealed a surprising result. The means 
of the time and variances were practically equivalent for both the pilot and nonpilot 
groups (Figure 21). 
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Figure 21. Recovery time statistics 
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According to an Anderson-Darling normality test, the reeovery times cannot be 
assumed to be normally distributed. However, the distribution shows a number of 
extreme outliers. We deeided to keep those data points in the dataset. 


7, Post-experiment Questionnaires 

We determined how often each participant chose a eertain answer option for eaeh 
question. This provided a first impression, after eompleting the entire experiment, about 
the subjective evaluation. The answers were determined in percentages as follows. For 
all, pilots and non-pilots, 40 percent, 50 percent/30 percent, means 40 percent of all 
participants chose this answer; 50 pereent of the pilots, 30 percent of the non-pilots 
(Table 3, Table 4). Because this experiment did not conduct any workload measurements, 
the questions about workload were not evaluated. 


Table 3. Results of post-experiment questionnaires: orientation task 


Question 

Answer-Options 

In general, how do you judge the 
difficulty of evaluating the current 
attitude of the aircraft? 

Very 

easy 

5% 

(10% /0%) 

Somewhat easy 

35% 

(40% /30%) 

Border 

line 

15% 

(10%/20%) 

Somewhat 

complicated 

45% 

(40%/50%) 

Very difficult 

0% 

(0%/0%) 

In general, how do you judge 
difficulty of using the traditional 
flight instruments for this task? 

Very 

easy 

5% 

(10% / 0%) 

Somewhat easy 

15% 

(30%/0%) 

Border 

line 

20% 

(10%/30%) 

Somewhat 

complicated 

50% 

(40%/60%) 

Very difficult 

10% 

(10%/10%) 

In general, how do you judge the 
difficulty of using the WEBER-Box 
for this task? 

Very 

easy 

40% 

(50%/30%) 

Somewhat easy 

60% 

(50%/70%) 

Border 

line 

0% 

(0%/0%) 

Somewhat 

complicated 

0% 

(0%/0%) 

Very difficult 

0% 

(0%/0%) 

Did you feel better aware of the 
spatial orientation of the airplane 
with the WEBER-Box in comparison 
with the traditional flight 

instruments? 

Much less 

0% 

(0%/0%) 

Somewhat less 

0% 

(0%/0%) 

Border 

line 

0% 

(0%/0%) 

Somewhat 

more 

40% 

(50%/30%) 

Much more 

60% 

(50%/70%) 
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Table 4. Results of post-experiment questionnaires: reeovery task 


Question 

Answer-C 

>ptions 

In general, how do you judge the 
difficulty of the recovery task? 

Very 

easy 

10% 

(10%/10%) 

Somewhat easy 

35% 

(30%/40%) 

Border 

line 

25% 

(30%/20%) 

Somewhat 

complicated 

30% 

(30%/30%) 

Very difficult 

0% 

(0%/0%) 

In general, how do you judge the 
difficulty of using the traditional flight 
instruments for this task? 

Very 

easy 

5% 

(10% / 0%) 

Somewhat easy 

20% 

(20% 20%) 

Border 

line 

35% 

(30%/40%) 

Somewhat 

complicated 

35% 

(40%/30%) 

Very difficult 

0% 

(0%/0%) 

In general, how do you judge the 
difficulty of using the WEBER-Box for 
this task? 

Very 

easy 

20% 

(10% 30%) 

Somewhat easy 

55% 

(70% 40%) 

Border 

line 

15% 

(10%/20%) 

Somewhat 

complicated 

10% 

(10%/10%) 

Very difficult 

0% 

(0%/0%) 

Did you feel better aware of the spatial 
orientation of the airplane with the 
WEBER-Box in comparison with the 
traditional flight instruments? 

Much less 

5% 

(10% / 0%) 

Somewhat less 

5% 

(10% / 0%) 

Border 

line 

0% 

(0%/0%) 

Somewhat more 

30% 

(10%/50%) 

Much more 

60% 

(70%/50%) 

In general, how helpful or unhelpful 
was the WEBER-Box in aiding your 
management of the airplane in 
comparison with the traditional flight 
instruments? 

Very 

unhelpful 

0% 

(0%/0%) 

Somewhat 

unhelpful 

5% 

(10% / 0%) 

Border 

line 

5% 

(10% / 0%) 

Somewhat 

helpful 

30% 

(20%/40%) 

Very helpful 

60% 

(60%/60%) 

How helpful or unhelpful was the 
WEBER-Box in aiding your task in 
recovering from an unusual attitude in 
comparison with the traditional flight 
instruments? 

Very 

unhelpful 

5% 

(10% / 0%) 

Somewhat 

unhelpful 

0% 

(0%/0%) 

Border 

line 

10% 

(10%/10%) 

Somewhat 

helpful 

25% 

(20%/30%) 

Very helpful 

60% 

(50%/70%) 


The diffieulty of the tasks was experieneed as “slightly difficult” to “somewhat 
easy” to accomplish. The answers on the post-experimental questionnaires showed a 
strong tendency toward the helpfulness of the proposed display design. Though one pilot 
(participant ID 108) responded strongly against the design, his objective results showed 
that he took notable advantage of the new display design, improving his orientation time 
in task one by 4.0 seconds. He was also able to recover the aircraft 3.8 seconds faster. 
That large discrepancy between subjective judgment and objectively measured results 
was one isolated case in our study. However, it shows that subjective impression 
measures and objective performance measures may diverge. In particular, pilots may 
have strong biases toward the display systems they were trained on and/or have used 
throughout their aviation careers. The questionnaires show, however, that most 
participants, pilots and non-pilots—over 80 percent—appreciated the proposed design 
and felt that it supported their spatial orientation tasks. 
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C PRELIMINARY ANALYSIS OF DISPLAY EFFECTS 
1. Effects on Orientation Time and Accuracy 

When we eompare the distributions of the two designs, the proposed and the 
traditional, the main differenees are obvious. The new design aehieves a mueh shorter 
orientation time. In addition, the varianee of the data is deeeased by using the proposed 
design. The differenee between the means is 4.36 seeonds and the standard deviations 
differ by 1.21 seeonds (Table 5). Furthermore, the difference in the means is greater than 
the previously determined threshold of 2.0 seconds. 


Table 5. Means and standard deviations: orientation time 


Level 

Number 

Mean 

Std Dev 

Std Err Mean 

Old Design 

120 

7.037 

2.255 

0.206 

New Design 

120 

2.671 

1.049 

0.096 


A comparison of the means graph and the distribution graph does not reveal any 
information about individual differences between the participants. However, when we 
compare the individual differences in the data-plot with the corresponding individual 
means (Figure 22), the individual differences are noticeable as different slopes. This 
graph was generated by the statistical software JMP®, version 5.1.2, using the analysis 
function “Fit Y by X.” Figure 22 shows that all the participants improved their 
orientation times with the proposed display. 
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Figure 22. Box-plot of individual changes in orientation time 

2, Effects on Recovery Time 

Similar observations as those for the orientation time can be made for the 
recovery time. The distributions differ noticeably in the means and variances of the data. 
The distributions show the differences in the means of recovery times for all participants. 
The means differ by 4.54 seconds; the standard deviations differ by 1.22 seconds (Table 
6). Furthermore, the difference in the means is greater than the previously determined 
threshold of -2.0 seconds. 


Table 6. Means and standard deviations: recovery time 


Level 

Number 

Mean 

Std Dev 

Std Err Mean 

Old Design 

120 

10.382 

3.382 

0.309 

New Design 

120 

5.844 

2.157 

0.197 


As noted before, a comparison of the means and the distribution does not reveal any 
information about individual differences. 
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Figure 23. Box-plot of individual changes in recovery time 


Individual differences are shown in the data-plot for all participants with 
corresponding individual means (Figure 23). All participants improved their recovery 
time performance with the new display. 


D, HYPOTHESIS TESTING AND RESULTS 

1, General Statistical Methodology 

To investigate the influences of the designs on orientation time, we conducted an 
extended, paired t-test (Montgomery & Runger, 2003). Participants’ IDs were used as a 
blocking factor. We conducted a stepwise regression analysis to eliminate confounding 
factors that were not influential in our model (Carrol & Ruppert, 1988). The potential 
predictor variables were; instrument design (a categorical variable), the main factor; the 
trial number to evaluate possible learning effects; flight hours for pilots; and participants’ 
ages. These variables were considered as independent, but they all may have had an 
influence on the outcome. The participants were handled as a blocking factor for the 
regression model. The response variables were: orientation time, pitch-and-roll error, and 
recovery time. 
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Only the display design had a signifieant influenee as a predietor variable for all 
response variables. The number of flight hours was a statistieally signifioant predietor of 
reeovery time F(l,l 18) = 124.07, p < 0.005, though the graph showed that it was nothing 
more than a caleulated value, with no praetieal relevance (Figure 24). The “flight hours” 
effect was mainly based on one outlier data-point for a participant with a very low 
number of flight hours. Linear regression determines linear correlations in a data set, 
even when the relationship is ambiguous or nonlinear (Montgomery & Runger, 2003). 
Thus, as an independent variable, flight experience contributed very little to explain the 
data (change in R = 0.06). Therefore, we were able to ignore this factor. 

We conducted stepwise regression for the subgroups, pilots and non-pilots, with 
similar results. 



Figure 24. Scatter-plot of flight hours vs. recovery time 


2. Primary Hypothesis 

a. Orientation Time 

The first primary hypothesis was that participants will be quicker in 
assessing their spatial situations using the proposed design. We generated a null 
hypothesis that there would be no difference between the orientation times of the two 
designs. Hence, the null hypothesis Ho is =[//2 “A] = 0 • The difference of 
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orientation times is zero. An alternative hypothesis was that the orientation time using the 
proposed design is shorter than the orientation time using the traditional design. Henee, 
Ha is < 0 ; the difference is negative. 

The results of the paired t-test were calculated for a two-sided t-distribution, were 
the notation ty determines the t-value for v = degrees of freedom. The paired t-test 
enabled us to reject the Ho at a level of significance of a = 0.05, by providing a test 
statistic of t 2 i 9 = 18.45, p < 0.0001, for the design as the major predictor (Figure 25). The 
detailed statistical data of the paired t-test is shown in Table 7. 


Table 7. Statistics of differences in orientation time 


Time Difference 

4.366 sec. 

t Ratio 

18.447 

Std Err Diff. 

0.236 sec. 

DL 

219 

Upper CL Diff. 

4.832 sec. 

Prob > \t\ 

0.0001 

Lower CL Diff 

3.899 sec. 

Prob > t 

0.0001 

Confidence 

0.95 

Prob < t 

0.9999 
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Figure 25. Connected means of orientation time 

For the orientation time task, the difference between the means of the two 
display groups was negative 4.366 seconds. Thus, it took an operator, on average, 2.5 
times longer to interpret the traditional instruments. 

Figure 26 shows the distance between the two sets of orientation time for the 
intrinsically linear regression model. 
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Figure 26. Statistical distance of orientation time means (sec) 
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The center shape denotes the distributions if Ho is supported. The line to the 
right determines the actual difference in the means. The statistics of the absolute 
values for the orientation time task were as follows: 

Traditional design: p = 7.04 seconds a = 2.48 seconds 

Proposed design: p = 2.68 seconds a = 1.13 seconds 


Similar statistical outcomes were observed for the two subgroups. Both pilots and 
non-pilots showed statistically different orientation times: 

Pilots: tio 9 = 16.311, p < 0.0001 

Non-pilots: tio 9 = 11.423, p < 0.0001 


Thus, pilots took slightly more advantage of the new design than non-pilots (Apo 
= 0.37 seconds). This result has no practical or statistical relevance, since being a pilot 
was a nonsignificant factor in the overall linear regression model. 

The value of 4.36 seconds mean time difference was more than twice the 
amount of our threshold value of 2.0 seconds for practical relevance. The 
primary hypothesis was that a person, using the proposed instrument design, would 
more quickly assess a spatial situation accurately than if he were using traditional 
instruments. Thus, this hypothesis is strongly supported by our statistical analysis (Table 
7). 

b. Recovery Time 

The second primary hypothesis was that the recovery time for both designs 
would be equal. We generated a null hypothesis, similar to our first primary null 
hypothesis, that there would be no difference between the reaction times of the two 
designs. An alternative hypothesis is that the reaction time using the proposed design is 
shorter than the reaction time using the traditional design. Hence, Ha is < 0; the 
difference is negative. 


80 



Again, to fulfill the model assumptions of normally distributed data 
(difference values), we had to transform the data set by applying the natural 
logarithm. We discuss this topic in depth in Chapter V (D.2.C.). 

The results of the paired t-test were calculated for a two-sided t-distribution, were 
the notation C determines the t-value for v = degrees of freedom. The paired t-test 
enabled us to reject the Ho at a level of significance of a = 0.05, by providing a test 
statistic with t 2 i 9 = 11.885, p < 0.0001 (Figure 27). The mean difference in recovery time 
in this experiment was negative 4.54 seconds. This means that it took an operator, on 
average, about 70 percent longer to recover the aircraft with traditional instruments. This 
value was more than 150 percent greater than the anticipated practically relevant time 
difference of positive 2.0 seconds (Table 8). Thus, our second primary hypothesis is 
robustly supported by the statistical analysis. The achieved statistical power was greater 
than 0.999. Hence, the probability of falsely rejecting a true hypothesis was less than 0.1 
percent. 


Table 8. Statistics of differences in recovery time 


Time Difference 

4.537 sec. 

t Ratio 

11.885 

Std Err Diff. 

0.382 sec. 

DL 

219 

Upper CL Diff. 

5.289 sec. 

Prob > \t\ 

0.0001 

Lower CL Diff. 

3.785 sec. 

Prob > t 

0.0001 

Confidence 

0.95 

Prob < t 

0.9999 
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Figure 27. Connected means of recovery time 


Figure 28 shows the distance between the two sets of orientation time for the 
intrinsically linear regression model. 



Figure 28. Statistical distance of recovery time means 

The center shapes denote the distributions if Ho is supported. The line to the 
right determines the actual difference in the means. The statistics of the absolute 
values were as follows: 

Traditional design: p = 10.44 seconds a = 2.37 seconds 

Proposed design: p = 5.83 seconds a = 1.84 seconds 
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A similar statistical picture was achieved by analyzing the two subgroups. Both, 
pilots and non-pilots, showed statistieally different recovery times: 

Pilots: tio 9 = 9.583, p < 0.0001 

Non-pilots: tio 9 = 7.439, p < 0.0001 

The analysis showed that pilots took slightly more advantage of the new design 
than non-pilots (ApD = 0.37 seeonds). This differenee has negligible statistieal or 
praetieal relevanee. However, the value of 4.54 seconds is greater by far than our 
threshold value of -2.0 seeonds for practical relevance. Henee, this seeond primary 
hypothesis is supported: use of the proposed display resulted in signifieantly shorter times 
for aireraft reeovery. The ealeulated statistieal power was greater than 0.999. Thus, the 
probability of rejeeting a eorreet hypothesis was less than 0.1 pereent. 

c. Model Adequacy 

To support the validity of the statistieal findings, we investigated whether the 
assumptions were met to apply a paired Mest. Those assumptions ineluded, foremost, that 
the delta values Pd were normally distributed (the normality assumption). The normality 
assumption was neeessary for the error rates we were willing to accept when making 
decisions about the proeess (Montgomery & Runger, 2003). 

Many statistieal tools are possible for model validation: one tool for proeess- 
modeling applieations is graphieal data analysis. Graphieal methods have an 
advantage over numerieal methods for model validation beeause they readily illustrate 
a broad range of eomplex aspeets of the relationship between the model and the data. 

To test the adequaey of the model, we plotted the distribution of the 
differenee of means to determine whether it supports the assumption of being 
normally distributed. The results show that the normality assumption holds for both 
orientation time and recovery time (Figure 29). 
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Figure 29. Model adequacy 


The Anderson-Darling test of normality suggests not rejecting the null 
hypothesis that the data is normally distributed, with p = 0.209 for orientation time 
and p = 0.753 for recovery time (Montgomery & Runger, 2003). Hence, this test 
supports the 
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assumption that the data was normally distributed; it also supports the previously stated 
model assumptions for applying a paired t-test. 


3, Secondary Hypothesis 

As stated previously, our seeondary hypothesis was that the angular precision, in 
terms of the pitch and roll of the assessed spatial orientation, was at least at the same 
level for both instrument designs. To prove or disprove this hypothesis, we conducted a 
nonparametric test on the error values of pitch and roll, since the model assumptions of 
normally distributed data did not support the use of a regular Mest. We summed the 
angular errors of each participant and calculated the difference for the two displays. Then 
we calculated the median of these differences: 
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a. Accuracy of Pitch 

The box-plot of the data suggests a significant reduction of the pitch errors by the 
new display design (Figure 30). We determined the null hypothesis as: there was no 
difference between the pitch errors of the two designs. Hence, the null hypothesis Ho 
is //, = //j ? or = 0 . An alternative hypothesis is that the pitch-and-roll error of the new 

design is smaller than that of the conventional design. Hence, Ha is //, > //j or > 0 . 
The level of significance is determined by an a of 0.05. 
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Doiiplot of Errors i n Pitch 



Figure 30. Box-plot of errors in pitch (all participants). 

We conducted a nonparametric test to evaluate whether there was a 
significant difference in the angular errors for pitch. We summed the angular errors per 
participant and applied a Wilcoxon Signed Rank Test to those sums (Montgomery 
& Runger, 2003). The test statistic is iv = min(w*, w') where w'^ and w' are the sums of 
the positive and negative ranks of the differences. 

Since a = 0.05 and n = 20, the critical value is >^* 0.05 = 60. We will reject Hq; go = 
0 if IV < 60. The lest statistic suggested rejecting the null hypothesis with a p < 0.05, since 
iv"^ = 1 is less than the critical value w*o.o 5 = 60. Therefore, the data does strongly support 
the hypothesis that there would be a significant difference in pitch error between the 
population medians for the traditional and the proposed display design. 

Wilcoxon Signed Test and Cl of Pitch-Errors: 


IV^ w' 

W*o.05 

n 

Median Error 

1 209 

60 

20 

-52.50° 


Lower Cl (0.95) = -90.00° 

Upper Cl (0.95) = -30.00° 

The analysis for pilots and non-pilots shows a similar picture. The 

difference in the angular errors was significant for both subgroups as well. 
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b. Accuracy of Roll 

The data plot shows a significant decrease of the roll errors for the new design 
(Figure 31). Similar to the pitch error evaluation, we determined the null 
hypothesis as: there would be no difference between the roll errors of the two designs. 
Hence, the null hypothesis Ho is jLi^= jii^, ox= 0 . An alternative hypothesis is that the 
pitch-and-roll error of the new design is smaller than that of the conventional design. 
Thus, Ha is yi, > yij! oi" Ad > 0 • 



Figure 31. Box-plot of errors in roll (all participants) 

We applied the same statistical methodology to analyze the statistical 
significance for the roll error that we used to evaluate pitch errors. Applying 
the Wilcoxon Signed Ra nk Test resulted in a P-value < 0.05. 

The test statistic is w = min(w^, w'), where w'^ and w' are the sums of the 
positive and negative ranks of the differences. 

Since a = 0.05 and n = 20, the critical value is >^* 0.05 = 60. We will reject Hq: go = 
0, if IV < 60. The test statistic suggested rejecting the null hypothesis with a p < 0.05, 
since = 3 is less than the critical value w*o.o 5 = 60. 
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Wilcoxon Signed Test and Cl of Roll Errors: 


w"" w' 

W*0.05 

n 

Median Error 

3 207 

60 

20 

-180.0° 


Lower Cl (0.95) = -270.00° 

Upper Cl (0.95) = -90.00° 

The analysis of spatial judgment for the subgroups of pilots and non-pilots shows 
a similar pieture. The improvement regarding the angular errors was signifioant for both 
subgroups as well. 


c. Extreme Errors 

We were interested in how frequently extreme errors in spatial judgment 
oecurred, sinee those might lead to aircraft crashes. For errors in pitch, Table 9 shows the 
number of errors for judgments in error by at least 30 . 


Table 9. Number of extreme errors for the pitch angle (> 30°) 



Pilots 

Non-Pilots 

All Participant 

Traditional design 

13 

12 

25 

Proposed design 

1 

2 

3 


Table 10 shows the number of errors of wrong estimates of at least 90 
degrees for roll errors. The number of extremely misjudged roll angles decreased 
from eleven with the traditional display to zero with the proposed display. 
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Table 10. Number of extreme errors for the roll angle (> 90°) 



Pilots 

Non-Pilots 

All Participant 

Traditional design 

5 

6 

11 

Proposed design 

0 

0 

0 


This data shows that extreme misjudgments were radieally reduced by use of the 
proposed design (Figure 32). The number of extremely misjudged pitch angles was 
reduced by 85 percent and the number of extremely misjudged roll angles was reduced 
to zero. 



Figure 32. Number of extreme angular errors (all participants) 
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VI. CONCLUSIONS 


A. OVERALL ASSESSMENT OF THE STATISTICAL RESULTS 

The statistical analysis supports the effectiveness of the proposed display design 
in all aspects. The design strongly promotes spatial awareness in 3D orientation tasks. 
The time it takes to assess a spatial situation decreased significantly. The accuracy of 
spatial judgment improved. Judgment errors were minimized and extreme errors were 
almost eliminated. 

The experiment setup favored the traditional flight-instrument design. Pilots’ 
training usually includes hundreds of hours of training with a traditional flight-instrument 
layout. Moreover, the hundreds, or even thousands, of flight hours must have continued 
to enhance the pilots’ ability to read and interpret traditional instruments. In particular, 
roll- and pitch-angles are displayed separately and distinguished by traditional 2D 
instrument designs (Figure 33). On the post-experiment questionnaires, pilots judged the 
difficulty in using traditional instrumentation for this task in 40 percent of the cases “very 
easy” or “somewhat easy.” These results represent the pilots’ confidence in their own 
ability to read and interpret common flight instrumentation. The numbers for non-pilots 
were similar (Table 3 and 4). In contrast, the proposed design shows a three-dimensional 
picture of the same spatial situation. Therefore, a participant had to decompose the pitch- 
and roll-angle into discrete values. However, it is important to note that the all 
participants were only able to train with the new instrumentation for about ten minutes. 
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Figure 33. Gyro attitude indieator instrument 
(Sereenshot of X-Plane'^’^ flight simulation) 


The proposed display design showed superiority in the statie spatial awareness 
task. All partieipants were able to improve their speed in evaluating a spatial situation 
signifieantly. With praotieally no training, the partieipant intuitively understood the way 
the spatial orientation was represented. The mean time it took non-pilots to assess the 
situation eorreetly using traditional instrumentation was 273 pereent longer than for the 
proposed design and 252 pereent longer for pilots. Thus both groups inereased their 
performance by over 250 percent using the proposed display design. 

More important, extreme misjudgments, which may lead to accidents, were 
almost eliminated. The participants’ self-assessments, as reflected in the post-experiment 
questionnaires, strongly support the statistical findings. However, the participants’ 
opinions about how helpful the new instrument design was in solving the static spatial 
orientation task show a 100 percent positive evaluation (Figure 34). That is, 40 percent of 
the participants found the new design very easy to use; 60 percent, “somewhat easy” to 
use. 
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Figure 34. Percentage self-assessment of difficulty to use new design (Orientation 

Task) 

More importantly, 60% of the participants found that their spatial awareness 
improved “much more” and 40% felt “somewhat more” spatial awareness in comparison 
with the old instrumentation (Figure 35). 



■ueh hm 


Figure 35. 


Percentage self-assessment of improvement of spatial awareness 
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Figure 36 shows that the level of improvement was different for every individual. 
The steeper the line, the more an individual took advantage of the new design. It seems 
that the pilots were able to improve their speed slightly better than the non-pilots did. 




Figure 36. Individual performance for orientation task (Pilots vs. Non-Pilots) 


Two questions remain regarding second experiment: a) Will the proposed design 
influence the flight task? b) How will the new design be accepted by the participants? 
The results of the recovery task show that the new design was widely accepted by the 
participants. As seen in Figure 37, almost all the participants improved their recovery 
time using the new instrument; only one pilot achieved equal performance. No participant 
did worse. 
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Pilots 


Non-Pilots 




Figure 37. Individual performance recovery task 


There were no statistical differences between the pilot and non-pilot group in their 
performance in the recovery task when using the new instrument. Both groups were able 
to decrease the variance of time needed for their recovery maneuvers. On average, 
recovery time performance improved by over 70 percent. 

The post-experiment questionnaires support these findings. When asked how easy 
the new design was to use for accomplishing the recovery task, 75 percent of the 
participants evaluated it as “very easy” or “somewhat easy” (Figure 38). 
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Figure 38. Percentage self-assessment of difficulty to use the new design (Recovery 

Task) 
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According to the questionnaires, 90 pereent of the partieipants judged that the 
proposed design supported their spatial awareness during the reeovery task. This result 
eonfirms our statistieal findings (Figure 39). 



LIni mora 

Figure 39. Pereentage self-assessment of improvement of spatial awareness 

(Reeovery Task) 

We observed a similar outeome for the questions about how mueh the new design 
aided the recovery task. Approximately 90 percent of the participants responded 
positively. These results support the hypothesis that the new design would aid in 
managing the aireraft and be helpful in the reeovery task. 


B. DISCUSSION AND CONCLUSIONS 

The proposed design was developed to support operators of aireraft and UAVs in 
extreme situations. Our main goal was to provide an intuitive way to improve the level of 
operators’ spatial awareness in ambiguous eonditions. The design implements the 
principles and guidelines for human-centered interfaee design. In particular, it draws on 
the prineiple of the mental model by introducing an exocentric view of a virtual avatar in 
an abstraet three-dimensional virtual world. 
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The two experimental tasks, in whieh partieipants judged their own spatial 
orientation and reeovered their aireraft from unusual attitude, tested the design in a static 
and dynamic environment. The outcome was remarkably clear. Using a flight instrument 
according to the proposed design, a performance gain of over 150 percent was found for 
the time to assess a spatial situation and a 70 percent performance increase was found for 
the recovery task. Both effects were on a level of significance a = 0.05. 

The pilots, in particular, were able to quickly understand the underlying 
advantages of the new instrument. Most of the pilots conducted their initial recovery 
maneuvers guided by the spatial information provided by the new instrument design. 
After their first few flight maneuvers, they switched back to the traditional 
instrumentation to conduct the precise final maneuvers. This made their flight maneuvers 
more efficient and shortened the overall recovery time. 

This positive picture was supported by the comments and responses on the post- 
experimental questionnaires. An overwhelming majority of the participants found the 
“WEBER-Box” very helpful and intuitively understandable. The participants felt that the 
“WEBER-Box” improved their spatial awareness and made it easier for them to 
accomplish their tasks. 

Often the pilots’ maneuver time was not significantly longer than the time to 
assess the spatial situation in the previous experiment. There are two possible 
explanations for this: (1) dynamic instruments (during the second task) provide much 
more information than a static picture (during the first task). People can better judge their 
position, when they observe a dynamic, changing display; (2) the operators may have 
started their recovery maneuvers without sufficient spatial awareness. They simply did 
something, observed the reaction of the gauges, and built their recovery strategy by 
minimizing their error, rather than drawing on an initial strategy. 

It was surprising that the pilot and non-pilot results were so similar. The 
differences were minor. We explain this phenomenon on the basis of the tests not being 
overly demanding compared to a real flight recovery task. Another aspect to consider is 
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that the underlying eognitive proeesses involved for both are basieally the same. Thus, 
both pilots and non-pilots eould take advantage of the proposed design. 

The proposed design proved to be easily and intuitively understandable. The 
eoneept showed supremaey over traditional designs even for pilots, who have aeerued 
thousands of hours working with traditional flight instruments. However, training and 
integration proeedures for implementing the new design have yet to be developed and 
tested. Only then will the proposed design’s full potential be evident. 

Many of the underlying interaetions and effeets of the various variables involved 
in the proposed display design were not eompletely investigated. Questions about design 
improvements and praetieal applieations also need to be answered. However, by applying 
human-eentered design prineiples, we were able to design an effieient tool to support 
spatial awareness in a 3D orientation task. The design was aeeepted and appreeiated by 
both the pilot and the non-pilot partieipants and was proven to be benefieial in eomplex 
spatial awareness tasks. 
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VII. RECOMMENDATIONS AND FURTHER STUDIES 


A. FUTURE APPLICATIONS 

We see a potential for the applieation of this design in aviation safety. Over the 
last few deeades, eoekpit design has improved substantially and display teehnology has 
enabled the implementation of three-dimensional displays. The “WEBER-Box” eould 
eontribute to inereased flight safety, espeeially in eases of pilot spatial disorientation. 

Furthermore, given the inereasing demand for UAV operation by non-pilots, who 
have not been trained in the use of traditional flight instruments, the WEBER-Box is 
potentially applieable for the eontrol of UAVs. The proposed design eould be helpful in 
the design of new UAV eontrol interfaees. It might also be possible to apply the display 
design to faeilitate the task of eontrolling multiple UAVs by a sole operator. 

In addition, sinee the proposed display design is intuitively understandable, it may 
help flight students to better understand traditional flight instruments. Thus, its 
integration into flight training would be benefieial. 


B. FUTURE RESEARCH 

This study’s goal was to design, develop, implement, test, and evaluate a novel 
display design, but the eondueted experiments raised more questions than they answered. 
Those questions are open to future researeh. Foremost among them are questions about 
design improvements and the influenee of changes and simplifications. What is the 
optimal size? How much can we simplify the aircraft model? Should we integrate scales 
to make it a complete flight instrument? What is the right place for it in a cockpit layout? 
How often does an operator look at this instrument during a recovery task? 

In addition, since we did not evaluate the design’s potential affect on the 
operator’s mental workload, research might investigate that issue. We also did no 
analysis of the altitude representation or the various possible implementations of warning. 

Finally, research should be done to investigate whether the design’s 
representation of spatial orientation is applicable to other orientation tasks. 
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APPENDIX A: IRB APPLICATION AND APPROVAL 



Michael E. McCauley, Ph.D. 
Operations Research Department 
Glasgow Hall 
Naval Postgraduate Sehool 
Monterey, California 93943 


To; Protection of Human Subjects Committee 

Subject: Application for Human Subjects Review for 

Use of Avatars to Support and Enhance Spatial Awareness in 3D- 
Orientation-T asks 


1. Attached is a set of documents outlining a proposed experiment to be conducted over 
the next year to support the thesis of Major Axel Weber, GE. 

2. We are requesting approval of the described experimental protocol. An experimental 
outline is included for your reference that describes the methods and measures we 
plan to use. 

3. We include the consent forms, privacy act statements, questionnaires, and briefing 
forms we will be using in the experiment. 

4. We understand that any modifications to the protocol or instruments/measures will 
require submission of updated IRB paperwork and possible re-review. Similarly, we 
understand that any untoward event or injury that involves a research participant will 
be reported immediately to the IRB Chair and NFS Dean of Research. 


Michael E. McCauley 
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APPLICATION FOR 
HUMAN SUBJECTS REVIEW 
_ (HSR) _ 


HSR NUMBER 


(to be assigned) 


PRINCIPAE INVESTIGATOR(S) (Full Name, Code, Telephone) 

Michael E, McCauley, Ph,D., Monterey, CA 93943, Phone: 831 656-2191 
(Thesis Major Axel Weher, GE) 


APPROVAL REQUESTED [ X ] New [ ] Renewal 


EEVEE OE RISK [ ] Exempt [ ] Minimal [ ] More than Minimal 


Justifieation: 3D-Virtual Environment (fixed-base flight simulator) 

WORK WIPE BE DONE IN (Site/Bldg/Rm) 

NPS, Watkins Hall, Rm 212B 

ESTIMATED NUMBER OP DAYS TO 
COMPLETE 

60 

MAXIMUM NUMBER OP SUBJECTS 

55 

ESTIMATED EENGTH OP EACH 
SUBJECT’S PARTICIPATION 

2 hours (30 min active in simulator ) 


SPECIAL POPULATIONS THAT WILL BE USED AS SUBJECTS 

[ ] Subordinates [ ] Minors [ X ] NPS Students [ ] Speeial Needs (e.g. Pregnant women) 


Speeify safeguards to avoid undue influenee and proteet subjeet’s rights; none 


OUTSIDE COOPERATING INVESTIGATORS AND AGENCIES 


- none- 

_ [ ] A eopy of the eooperating institution’s HSR deeision is attaehed. _ 

TITEE OE EXPERIMENT AND DESCRIPTION OP RESEARCH (attaeh additional sheet if 
needed). Methodology attached 


I have read and understand NPS Notiee on the Proteetion of Human Subjeets. If there are any 
ehanges in any of the above information or any ehanges to the attaehed Protoeol, Consent 
Porm, or Debriefing Statement, I will suspend the experiment until I obtain new Committee 
approval. 

SIGNATURE_ 

DATE 
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From: Crowson, Jeff USA 

Sent: Tuesday, July 19, 2005 5:41 PM 

To: Weber, Axel DEU 

Subject: IRB Approval 

MAJ Weber, 

Your study, entitled "Use of Avatars to Support and Enhance Spatial Awareness in 3D- 
Orientation-Tasks" has been found to meet the criteria for an "Expedited" Review. I have 
consulted with the NPS IRB Medical Advisor, LTC Eric Morgan, MC, USA, and he has advised 
me that any potential side effects associated with the use of the flight simulator would be 
expected to be minimal. As I have advised you verbally, the only procedure we request you 
follow is to have all subjects agree to wait 20 minutes before driving after participating. As in all 
studies, if any subject reports not feeling well or experiences an untoward reaction, I ask that you 
report that incident to me and to the Dean of Research, Dr. Leonard Ferrari. That said, you are 
free to begin the study. You will receive a formal letter stating the above in writing. 

Please advise the IRB Chair when you have completed the study, and also turn in all signed 
subject consent forms to the IRB office for storage (they will be retained for 5 years, at which 
point they will be destroyed). Good luck with your study! 

v/r 

Jeff Crowson, Ph.D. 

Chair, Institutional Review Board 
Naval Postgraduate School 
Monterey, CA 
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APPENDIX B: EXPERIMENT PROTOCOL 


PROTOCOL: 

DAILY FLIGHT-SIMULATOR ENVIRONMENT SIMULATOR STARTUP 

Note: Use this protoeol when starting up the simulator eaeh day. 


Starting the CAVE-Projection: 

1. Switch on the computer CAVEMAN-1, CAVEMAN-2 and CAVEMAN-3 

2. Switch on projector 1, 2 and 3 by using the remote control 

3. Switch on the three-panel monitor 

4. Toggle between the CAVEMAN computer by pressing the SCROEE/EOCK- 
button twice and press 1 for CAVEMAN-1, 2 for CAVEMAN-2 and 3 for 
CAVEMAN-3 

5. Eogin at all CAVEMAN-computer by pressing ALT DEE CTRL. Use password 
“caveman” and press ENTER 

6. Start on every CAVEMAN-computer the application “IG 2004 12 08” on the 
desktop by double-clicking 


Starting the Flight Simulator: 

1. Switch on the computer EEIGHT MODEE and HOST 

2. Switch on the TET monitor 

3. Toggle between the EEIGHT MODEE and HOST computer by pressing the 
SCROEE/EOCK-button twice 

4. Eogin at both computer by pressing ALT DEE CTRL. Let the password empty 
and press ENTER 

5. Toggle to the HOST-computer by pressing the SCROLL/LOCK-button twice and 
start on the desktop the following programs by clicking the icons twice by the 
mouse: 

a. TRCCustom => wait for the cockpit gauges to adjust 

b. Erik Dev EAA => go to Tab “Database” and chose “29Plams” 

=> go to Tab “Environment”, set Visibility to 0.05 and 
Time to 10:00 AM 

6. Toggle to the EEIGHT MODEL -computer by pressing the SCROLL/LOCK- 
button twice open the folder “Experiment Major Weber” on the desktop 

7. Switch on the computer CHALLENGER, the 17” TET monitor on the 
experimenter desk and the 10” TFT monitor at the top of the cockpit 

8. Login at the computer by pressing ALT DEE CTRL. Let the password empty 
and press ENTER 

9. Make sure that the 10” TFT monitor extends the desktop 


111 



10. Open the “Loeal Area Network” ieon in the task bar using the RIGHT mouse 
button 

11. In the following menu ehose STATUS, eliek on the SUPPORT-tab and write 
down the IP-address (e.g. 131.120.151.6) 

12. Start on the desktop the following programs by elicking the ieons twiee by the 
mouse: 

a. “UDP Refleetor” => type in the dialog box the IP-address of the 
“CHALLENGER” eomputer und press ENTER 

b. “X-Plane Experiment” 

13. After “X-Plane” started, 

■=> ehoose menu “Eoeation” => seleet “Plaee Aireraft 
by Airport” => chose “Twenty-nine Palms EAE” 

=> click “Go to this Airport” 

■=> choose menu “Piles” => select “Eoad Scenarion” 

=> click on “Experiment.sit” 

■=> wait until the situation is loaded, than hit the “P” 
button of the keyboard to pause the simulation 

14. Place the TPT-monitor with now showing aircraft cockpit on the pillar in front of 
the cockpit construction and place it in a way that all instruments can clearly be 
seen from the pilot’s seat. 


Start the WEBER BOX 

1. On the CHAEEENGER -computer double click on the icon ’’Start Weber 
Box” icon -> in the window type in for the UDP port: 49001 and 0.5 for the 
speed. Now the WEBER-Box should appear. The experimenter can drag it to 
the 10” monitor for all experiments which demand the WEBER-Box. 

2. Double-click on the icon “Experimenter” to start the input consol. 

^ Type in the field “IP-Address” the number 
131.120.150.248 and click the button “set” right 
to the input field 

^ Type in the filed “ID” the number 16 and click 
the button “int” below the field 
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PROTOCOL: 

DAILY FLIGHT-SIMULATOR ENVIRONMENT SIMULATOR SHUTDOWN 

Note: If you are simply shutting down for the day, please follow this protoeol. 

1. Shutdown the projeetors by pressing the POWER-button twiee for eaeh projeetor 

2. Shutdown the CAVEMAN-1 to .. .-3 by toggling through the computers and go to 
the “Starf’-Menu on the Windows task bar. Now chose “Shut Down” and choose 
the menu point “Shut Down” and press ENTER. Turn the monitor off. 

3. Shutdown the CALEENGER computer by going to the “Starf’-Menu on the 
Windows task bar. Now chose “Shut Down” and choose the menu point “Shut 
Down” and press ENTER. Switch the power switch at the power distributor off 

4. Shutdown the ELIGHTMODEE and the HOST computer by toggling through the 
computers and go to the “Starf’-Menu on the Windows task bar. Now chose “Shut 
Down” and choose the menu point “Shut Down” and press ENTER. 

5. Go behind of the simulator cockpit and unplug the power cord for the gauges. 

PROTOCOL: 

PRE-EXPERIMENT BATTERY 

• Check if all forms are prepared and signed by the experimenter and the principle 
investigator 

• Escort participant to peroration desk 

• Administer Initial Questionnaires 

o Hand out the written experiment objectives and the experiment 
introduction. 

o Present Institutional “IRB Participant Consent Eorm”, the “Minimal Risk 
Consent Statement”, the “Privacy Act Statement” and let it sign by the 
participant. 

o Check if the participant’s ID number is noted on every page of the initial 
questionnaire and present it and let the participant sign it. 

• Administer Simulator Sickness Pre-Screening Questionnaire 

o Participants are NOT eligible for inclusion in any flight simulator 

environment simulation experiment if participant indicates either they are 
pregnant /potentially pregnant or is color-blind. 

o Participants are NOT eligible for inclusion in any flight simulator 
simulation experiment if participant selects more than two items of 
question one to eight on Simulator Sickness Pre-Screening Questionnaire. 

• Request if the participant has any further questions 
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PROTOCOL: 

SPATIAL AWARENESS EXPERIMENT 


Participant is led into Flight Simulator and shown ah equipment. 

Partieipant is instructed to sit in the operator/pilot compartment. 

Experimenter explains all flight instruments and controls inch the WEBER-Box. 
Experimenter starts the training scenario (the aireraft is airborne on a safe altitude 
and attitude) and allows the participant to fly for about 10 minutes. In this time 
the experimenter provides any help regarding explanations of the use of the flight 
instruments and controls. He must not refer to any of the later experiment tasks. 
Experimenter gives the participant the briehng for the first task (Recognition of 
spatial orientation of the aireraft) and makes partieipant familiar with the 
proeedures. He is now advised to read the task instructions for this task. 
Experimenter switches off the instrumentation monitor or switehes off the 
WEBER-Box respectively, depending if he begins with or without using the 
WEBER-Box. 

Experimenter asks participant if he/she understand everything and if he may start 
the experiment 

The first trial is a test trial; the test-trial sheet has to be used for it. 

Experimenter: 

o starts spatial awareness scenario 
o prepares the stop wateh 

Participant blindfolds him/herself by wearing opaque goggles 

Experimenter: Determines a set of pitch- and roll-angles from the task sheet “Test 

A” (“Test B” for the set of experiments) by 

o Typing the angle values in degrees in field “00” for the pitch and field 
“01” for the roll and elicks on the button “fit” below the field to store the 
values 

o Clicks the button “Send Data Package” to transfer the values to the flight 
simulator 

o Observes whether the flight simulator reacts as expected 
o If not, he ehecks again ah values and presses ah described buttons again to 
store the data 
Experimenter: 

o Stands direetly on the participants left side and plaees his left hand at the 
power button of the monitor and holds the stop wateh in his right hand 
o Cheeks if the partieipant is blindfolded 
o Tells the participant he may start to look at the instruments 
o In the Moment the participant takes away the goggles and start to look at 
the instruments, he starts the stop wateh 
o When the participant says “Stop!” he switches of the monitor and stops the 
stop wateh at the same time 
Experimenter: 

o Eet the participant draw his/her opinion about piteh and roll into the 
provided sehema on the evaluation sheet 
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o Writes the measured time into field next to the evaluation schema 
Experimenter starts over with the next experiment loop and a new set of pitch and 
roll angles until the participant finished six tasks (plus one test trial). 
Experimenter starts with the RECOVERY EXPERIMENT 


PROTOCOL: 

RECOVERY EXPERIMENT 

Participant sits still in the operator/pilot compartment. He is now advised to read 
the task instructions for this task. 

Experimenter restarts the flight simulation by exit the currently running program, 
restarting the program “X-Plane” as described and loading the experiment 
scenario. 

Experimenter explains briefly the use of flight instruments and controls inch the 
WEBER-Box and the basic knowledge of the order of flight maneuvers to recover 
an aircraft from unusual attitude (first correct the roll, than correct the pitch). 

The Experimenter: 

o Informs the participant that he has now a test trial for the recovery test 
o Gives the order to blindfold 

o Gives the command “Press the button”, which let the participant activate 
the flight simulation 

o Provides flight commands to bring the aircraft in a unusual attitude (Pitch 
and/or Roll > 90°) above secure altitude 
o When the aircraft is in a desired attitude, he presses the button “P” at the 
flight simulator keyboard to pause the simulation and note down the 
angular orientation on the evaluation sheet 
o During this moment he continuous providing flight commands to not let 
the participant know that the simulation is actually paused 
Experimenter: 

o Makes sure that the WEBER-Box is the active window (dark blue frame) 
o Places one finger at the “P”-button of the flight simulator keyboard (left 
hand) and one finger at the “S”-button experimenter keyboard (right hand) 
- the “P”-button reactivates the simulation and the “S”-button starts the 
automatic maneuver time evaluation 
o Gives the instruction “Start Recovery” 

o Presses both buttons at simultaneously when the participant removed his 
goggles and is able to control the aircraft 
Participant starts the recovery maneuver and holds the aircraft leveled ( ±10° in 
pitch and roll) for at least five seconds 
Experimenter: 

o Observes the WEBER-Box display and determines when the maneuver 
time appears at the bottom of the display 
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o Let the partieipant know that the trial is over 

o Notes down the maneuver time on the evaluation sheet (not for the test 
trial) 

• If the partieipant erashed the aireraft during his/her test trial, he/she has another 
test trial 

• Experimenter asks partieipant if there are any questions 

• Experimenter starts the real experiment trials using the same proeedure as in the 
test trial until six valid experiment trial are finished 

• Any erash of the aircraft does not count as a valid trial. 


PROTOCOL: 

POST EXPERIMENT 

• Participant is allowed to have a five minutes break 

• Reseat the participant and start with the block of experiments including the 
spatial awareness experiment and the recovery experiment. 

• After all tests are completed the experimenter: 

o Thanks the participant for his/her effort 
o Guides participant to a chair 

o Provides the “Post-Experiment Questionnaire” and asks participant to fill 
it out 

o Evaluates the angular errors of the SA-experiment and notes it down into 
the evaluation sheets 

o Instructs the participant to wait at least additional 20 minutes before he 
steers a vehicle 

o Records all results in the participant’s database. 

• Thank participant for participating in the experiment. 
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APPENDIX C: PARTICIPANT INFORMATION FORMS 


PARTICIPANT CONSENT FORM 

1. Introduction, You are invited to participate in a study of spatial awareness of virtual 
environments. With information gathered from you and other participants, we hope to prove 
the concept of the use of virtual avatars to support and enhance spatial awareness and improve 
orientation performance. We ask you to read and sign this form indicating that you agree to be 
in the study. Please ask any questions you may have before signing. 

2. Background Information. The Naval Postgraduate School MOVES Institute is conducting 
this study. 

3. Procedures. If you agree to participate in this study, the researcher will explain the tasks in 
detail. There will be two sessions with two different orientation tasks each. The execution 
phases will last approximately two hours total, during which you will be asked to accomplish a 
number of tasks related to spatial awareness while operating a virtual flight simulator. We will 
ask you to wear a blindfold during several phases of this study. Following the study you will be 
asked to fill out questionnaires during a 20-minutes rest period. 

4. Risks and Benefits. This research involves having a participant fly a 3D virtual flight 
simulator. If you have any cardiac risk factors (High blood pressure, smoking, diabetes, high 
cholesterol, previous heart problems), we request that you PLEASE INFORM THE 
EXPERIMENT ADMINISTRATOR AND YOU WILL NOT BE ABLE PARTICIPATE IN 
THE EXPERIMENT. Other risks include the possibility of post-experiment vertigo due to 
simulator sickness. This may result in symptoms similar to motion sickness, and is therefore 
advisable not to drive within one hour of completing the experiment. On rare occasions, 
individuals experience a slight feeling of malaise. If this occurs, you will remain at the 
simulator site until the symptoms subside. We will make arrangements for you to be 
evaluated by professional healthcare as needed. 

5. Compensation. No tangible reward will be given. A copy of the results will be available to 
you at the conclusion of the experiment period. 

6. Confidentiality, The records of this study will be kept confidential. No information will be 
publicly accessible which could identify you as a participant. 

7. Voluntary Nature of the Study. If you agree to participate, you are free to withdraw from the 
study at any time without prejudice. You will be provided a copy of this form for your records. 

8. Points of Contact. If you have any further questions or comments after the completion of 
the study, you may contact the research supervisor, Michael E. McCauley, Ph.D., Monterey, 

CA 93943, Phone: 831 656-2191 

9. Statement of Consent. I have read the above information. I have asked all questions and have 
had my questions answered. I agree to participate in this study. 


Participant’s Signature 


Date 


Researcher’s Signature 


Date 
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MINIMAL RISK CONSENT STATEMENT 

NAVAL POSTGRADUATE SCHOOL, MONTEREY, CA 93943 

Participant: VOLUNTARY CONSENT TO BE A RESEARCH 

PARTICIPANT IN: Use of Avatars to Support and Enhance 
Spatial Awareness in 3D-0rientation-Tasks 


1. I have read, understand and been provided "Information for Participants" that provides the 
details of the acknowledgments below. 

2. I understand that this project involves research. An explanation of the purposes of the 
research, a description of procedures to be used, identification of experimental procedures, 
and the extended duration of my participation have been provided to me. 

3. I understand that this project does not involve more than minimal risk. I have been informed 
of any reasonably foreseeable risks or discomforts to me. 

4. I have been informed of any benefits to me or to others that may reasonably be expected from 
the research. 

5. I have signed a statement describing the extent to which confidentiality of records identifying 
me will be maintained. 

6. I have been informed of any compensation and/or medical treatments available if injury 
occurs and is so, what they consist of, or where further information may be obtained. 

7. I understand that my participation in this project is voluntary; refusal to participate will 
involve no penalty or loss of benefits to which I am otherwise entitled. I also understand that 
I may discontinue participation at any time without penalty or loss of benefits to which I am 
otherwise entitled. 

8. I understand that the individual to contact should I need answers to pertinent questions about 
the research is Professor Michael E. McCauley, Principal Investigator, and about my rights as 
a research participant or concerning a research related injury is Prof Rudy Darken, MOVES 
Institute Chairman or the NPS IRB Medical Advisor, ETC Eric Morgan, MC, USA, Presidio 
of Monterey, (831) 242-7550, eric.morgan@NW.AMEDD.ARMY.MIE 


Signature of Principal Investigator Date 


Signature of Volunteer 


Date 
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PRIVACY ACT STATMENT 


NAVAL POSTGRADUATE SCHOOL, MONTEREY, CA 93943 
PRIVACY ACT STATEMENT 

1. Purpose: Spatial cognition data will be collected to enhance knowledge, and to 
develop tests, procedures, and equipment to improve the development of Virtual 
Environments. 

2. Use: Spatial cognition data will be used for statistical analysis by the Departments of 
the Navy and Defense, and other U.S. Government agencies, provided this use is 
compatible with the purpose for which the information was collected. Use of the 
information may be granted to legitimate non-government agencies or individuals by 
the Naval Postgraduate School in accordance with the provisions of the Freedom of 
Information Act. 

3. Disclosure/Confidentiality: 

a. I have been assured that my privacy will be safeguarded. I will be assigned a 
control or code number, which thereafter will be the only identifying entry on 
any of the research records. The Principal Investigator will maintain the cross- 
reference between name and control number. It will be decoded only when 
beneficial to me or if some circumstances, which is not apparent at this time, 
would make it clear that decoding would enhance the value of the research data. 
In all cases, the provisions of the Privacy Act Statement will be honored. 

b. I understand that a record of the information contained in this Consent Statement 
or derived from the experiment described herein will be retained permanently at 
the Naval Postgraduate School or by higher authority. I voluntarily agree to its 
disclosure to agencies or individuals indicated in paragraph 3 and I have been 
informed that failure to agree to such disclosure may negate the purpose for 
which the experiment was conducted. 

c. I also understand that disclosure of the requested information, including my 
Social Security Number, is voluntary. 


Name, Grade/Rank (if applicable) 
[Please print] 


Signature of Volunteer 


Date 
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APPENDIX D: QUESTIONNAIRES 


Initial Questionnaire 


Age 


Gender 

Male CH Female CH 


Are you experienced in aviation 



in any regard (inch playing Flight 
Simulation Games)? 

yesD 

NO □ 


Date: 


If you answered the last question 
with YES, please proceed HERE. _ 


Are you an aviator/pilot? 

YES 

□ 

NO 

□ 

Are you familiar with the basic set of 
flight instruments? 

YES 

□ 

NO 

□ 

Are you familiar with the basic flight 
controls to steer an airplane? 

YES 

□ 

NO 

□ 


Fixed 

□ 

Rotor 

□ 

Did you operate any of these aircraft?: 
Fixed-Wing (Jet/Propeller) 

Jet 

□ 

Propeller 

□ 

Rotor-Wing Aircraft 

UAV 

UAV 

□ 



How many flight hours do you have? 



Hours 


How many hours in a flight simulator do 
you have? 



Hours 


How long ago was your last flight? 

Years 


Monthtsl 





How long ago was your last use of a 

Years 


Month(s) 

flight simulator? 

Are you trained in procedures of 
recovery from unusual attitude? 

YES 

□ 

NO 

□ 
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Participant ID 


If you are an active/former 
aviator/pilot, please proceed HERE: 


Have you ever experienced any 
kind of Spatial Disorientation? 

YES 

□ 

NO 

□ 

More than once? 

YES 

□ 

NO 

□ 


Briefly describe the situation/s 
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SIMULATOR SICKNESS PRE-SCREENING QUESTIONNAIRE 


This study will require you to fly in a simulator. In the past, some participants have 
reported feeling little uneasy after participating in studies using the simulator. To help identify 
people who might be prone to this feeling, we would like to ask the following questions. 

1. Do you or have you had a history of migraine headaches? □ yes □ no 

If yes, please describe:_ 

2. Do you or have you had a history of claustrophobia? □ yes □ no 

If yes, please describe:_ 

3. Do you or have you had a history of motion sickness? Dyes □ no 

If yes, please describe:_ 

4. Any health problems which might affect operating the flight simulator? 

□ yes □ no 

5. Heart problems or heart attacks? 

□ yes □ no 

6. Suffer from epileptic seizures? 

□ yes □ no 

7. Any inner ear problems, dizziness, vertigo, or balance problems? 

□ yes □ no 


8. Are you currently taking any medications? □ yes □ no 

If yes, please describe:_ 

9. If you are a female, are you or is there a possibility that you might be pregnant? 

□ yes □ no 
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Post-Experiment Questionnaire 
(Spatial Awareness Task) 


Participant ID 


Question 

Answer-Options 

In general, how do you judge 
the difficulty of evaluating the current 
attitude of the aircraft? 

Very 

easy 

Somewhat 

easy 

Border 

line 

Somewhat 

complicated 

Very difficult 

In general, how do you judge 
difficulty of using the traditional flight 
instruments for this task? 

Very 

easy 

Somewhat 

easy 

Border 

line 

Somewhat 

complicated 

Very difficult 

In general, how do you judge 
the difficulty of using the WEBER-Box 
for this task? 

Very 

easy 

Somewhat 

easy 

Border 

line 

Somewhat 

complicated 

Very difficult 

Did you feel better aware of 
the spatial orientation of the airplane 
with the WEBER-Box in comparison 
with the traditional flight instruments? 

Much less 

Somewhat 

less 

Border 

line 

Somewhat more 

Much more 

Did having the WEBER-Box 
increase or decrease your monitoring 
demands in comparison with the 
traditional flight instruments? 

Greatly 

decreased 

Somewhat 

decreased 

Unaffected 

Somewhat 

increased 

Greatly 

increased 

Did having the WEBER-Box 
increase or decrease your overall 
workload in comparison with the 
traditional flight instruments? 

Greatly 

decreased 

Somewhat 

decreased 

Unaffected 

Somewhat 

increased 

Greatly 

increased 
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Post-Experiment Questionnaire 
(Recovery Task) 


^^articiganO^ 


Question 

Answer-Options | 

In general, how do you judge 
the difficulty of the recovery task? 

Very 

easy 

Somewhat 

easy 

Border 

line 

Somewhat 

complicated 

Very difficult 

In general, how do you judge 
the difficulty of using the traditional 
flight instruments for this task? 

Very 

easy 

Somewhat 

easy 

Border 

line 

Somewhat 

complicated 

Very difficult 

In general, how do you judge 
the difficulty of using the WEBER-Box 
for this task? 

Very 

easy 

Somewhat 

easy 

Border 

line 

Somewhat 

complicated 

Very difficult 

Did you feel better aware of 
the spatial orientation of the airplane 
with the WEBER-Box in comparison 
with the traditional flight instruments? 

Much less 

Somewhat 

less 

Border 

line 

Somewhat more 

Much more 

In general, how helpful or 
unhelpful was the WEBER-Box in 
aiding your management of the airplane 
in comparison with the traditional flight 
instruments? 

Very 

unhelpful 

Somewhat 

unhelpful 

Border 

line 

Somewhat 

helpful 

Very helpful 

How helpful or unhelpful was 
the WEBER-Box in aiding your task in 
recovering from an unusual attitude in 
comparison with the traditional flight 
instruments? 

Very 

unhelpful 

Somewhat 

unhelpful 

Border 

line 

Somewhat 

helpful 

Very helpful 

Did having the WEBER-Box 
increase or decrease your monitoring 
demands in comparison with the 
traditional flight instruments? 

Greatly 

decreased 

Somewhat 

decreased 

Unaffected 

Somewhat 

increased 

Greatly 

increased 

Did having the WEBER-Box 
increase or decrease your overall 
workload in comparison with the 
traditional flight instruments? 

Greatly 

decreased 

Somewhat 

decreased 

Unaffected 

Somewhat 

increased 

Greatly 

increased 
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Participant ID 


Comments/Suggestions/Opinions 

In General: 


Experiment Tasks: 


WEBER-Box: 
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Participant ID 


POST-EXPERIMENT SIMULATOR INDUCED DISCOMEORT 
QUESTIONNAIRE 

There is a very small risk associated with operating the flight simulator. An individual 
may experience feelings of dizziness and increased body temperature, which are often symptoms 
of a temporary condition called 'Simulator Induced Discomfort' (SID). 

To verify the extent of a SID occurrence, we are tracking the intensity of any discomfort 
experienced by those who use a flight simulator. 


During this experience in the flight simulator did you experience any feelings of 
discomfort? 


Eye Strain: 

none 

slight 

moderate 

severe 

Temperature increase: 

none 

slight 

moderate 

severe 

Dizziness: 

none 

slight 

moderate 

severe 

Headache: 

none 

slight 

moderate 

severe 

Nausea: 

none 

slight 

moderate 

severe 
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APPENDIX E: EVALUATION FORMS 


Participant iD: . 

WEBER BOX: YES □ NO □ 



Test 


• • • 



129 

































Recovery Task 


Participant ID: 


WEBER BOX: 

YES □ NO □ 



Trial 


Time to Recover 

Attitude 



In sec. 

(Pitch/Roll) 


1 



O 

O 

2 



O 

O 

3 



O 

O 

4 



O 

O 

5 



O 

O 

6 



O 

O 

WEBER BOX: 

YES □ NO □ 



Trial 


Time to Recover 

Attitude 



In sec. 

(Pitch/Roll) 


1 



O 

O 

2 



O 

O 

3 



O 

O 

4 



O 

O 

5 



O 

O 

6 



O 

O 
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APPENDIX F: TASK DESCRIPTIONS 


Spatial Orientation and Recovery Task 


Background 

You are the operator of a eombined rotor-/fixed-wing based Unmanned Aerial 
Vehiele (UAV) of type Eagle Eye. 



The Eagle Eye has a 
wingspan of 15.2 ft, is 17.9 
ft in length, is 5.7 ft high, 
and weighs around 2,000 
pounds (depending on 
payload). 

The UAV is 
equipped with the newest 
180° Eield-of-View eamera 
teehnology to support the 
operator. 
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The secure altitude is defined by 2500 feet. Below this altitude the risk of crashes 
and of being shot down by enemy fire soars. 


Experiment Goals 

The goal of this experiment is to test different flight instrument setups regarding 
their capabilities to support the operator in controlling the aircraft. Of particular interest is 
to determine operator workload and perception of unusual attitudes. 


Overall Experiment Schedule 


You will conduct two sets (blocks) of trials. The order in which you will 
experience the two blocks is assigned randomly. The only difference between the blocks 
is which set of flight instruments is used. 

After the task briefing, you will get the opportunity to become familiar with 
operating the flight simulator before you start with your assigned block of trials. After 
finishing your first block, you will have a short break to fill out a brief questionnaire, and 
then you will start with the second block of trials. 

After you have finished both sets of trials you will be de-briefed and asked to 
respond to a post-experiment questionnaire. 

Remember: 

• You are a volunteer - we truly appreciate your time and willing to participate at 
our experiment! 

• Do not put yourself under pressure, even though some tasks might be challenging. 

• Please complete all trials if possible, but you can quit the experiment at any time 
if you need to. 

• Due to the dynamics of the flight simulation displays, you might experience 
discomfort. Please do not hesitate to inform the experimenter and please do not 
hesitate to abort the experiment if the level of discomfort is too high for you. 

• All personal data will be handled confidentially and anonymously. 


The next page shows you the principle schedule of the experiment. 
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Experiment Schedule 


Intro and Initial 
Briefings 


Setup A 


Training Flight 
One 


Task One 


Task Two 




Break 

(optional) 

i 



1 


Setup B 


Training Flight 
Two 


Task One 


Task Two 


~r~ 

Debriefing 
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Task One 


The UAV is on a test flight and enters a sector of high crosswinds. 

To simulate the broken data-link, you will be exposed to a unknown attitude of 
the aircraft. 


Experiment Implementation 

You will “fly” blindfolded the UAV for 20 minutes or so. Periodically, the 
experimenter will give you the order to judge the current spatial attitude of the UAV. The 
simulation will freeze at this moment. 

You will than hear the command “Stop Flying”. You will immediately remove the 
opaque goggles and begin to evaluate your current spatial attitude using your flight 
instruments. When you feel you know what the current spatial attitude is like, say loudly 
“STOP”. At this moment the flight instruments will be switched off. 

Now you have to express the observed spatial attitude of the UAV relative to the ground. 
You will do this by determining pitch and roll which you will draw on an orientation 
chart. 

The time to accomplish this task and the accuracy of your spatial judgment will be 
measured. 

You will have one test trial to become familiar with procedure. 

Do you have any questions? 
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Task Two 


The UAV is on a test flight and enters a seetor of bad weather. 

The data-link eonneetion to the UAV was interrupted beeause of the weather 
eonditions (simulated by blindfolding). When the data-link connection is reestablished, 
your task is to recover and to stabilize the aircraft at a safe altitude. 

Recover and Stabilizing the UAV: 

The UAV is considered to be stabilized if the angles of pitch and roll are ±10 
degrees for at least 5 seconds and if the UAV flies at a safe altitude (> 2500 ft). 

Experiment Implementation 

You will control the UAV for about 20 minutes in this condition. The 
experimenter will periodically reset the simulation and start with a new UAV position to 
simulate multiple data-link interruptions. At the beginning of each task you will be 
blindfolded. You will fly the aircraft a short time by given commands. 

You remove the opaque goggles when you hear the command “Start Recovery”. Now 
you begin to take over the flight controls by pressing the button on the control stick. You 
have to maintain the leveled attitude for at least 5 seconds at a secure altitude. 

The time to accomplish this task will be measured automatically. 

You will have one test trial to become familiar with procedure. 

Do you have any questions? 
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APPENDIX G: EXPERIMENT DATA 


Participant 

ID 

Age 

Pilot? 

(0/1) 

Flight 

Hours 

Box? 

(0/1) 

started 

with 

Box? 

(0/1) 

Trial 

Orientation 

Time 

Error 

Pitch 

Error 

Roll 

Recover 

Time 

1 

21 

0 

0 

0 

0 

1 

5.2 

45 

0 

12.1 

1 

21 

0 

0 

0 

0 

2 

4.9 

60 

75 

18.5 

1 

21 

0 

0 

0 

0 

3 

5.2 

0 

30 

5.8 

1 

21 

0 

0 

0 

0 

4 

5.7 

0 

45 

7.8 

1 

21 

0 

0 

0 

0 

5 

5.4 

30 

90 

5.2 

1 

21 

0 

0 

0 

0 

6 

6.6 

30 

15 

5.9 

1 

21 

0 

0 

1 

0 

1 

2.2 

0 

0 

3.7 

1 

21 

0 

0 

1 

0 

2 

2.4 

0 

0 

4.4 

1 

21 

0 

0 

1 

0 

3 

3.1 

0 

0 

3.7 

1 

21 

0 

0 

1 

0 

4 

2.6 

0 

30 

8.2 

1 

21 

0 

0 

1 

0 

5 

2.5 

0 

0 

2.7 

1 

21 

0 

0 

1 

0 

6 

1.9 

0 

0 

5 

2 

31 

0 

0 

0 

1 

1 

10 

15 

75 

10.1 

2 

31 

0 

0 

0 

1 

2 

25.1 

0 

0 

20.9 

2 

31 

0 

0 

0 

1 

3 

13.2 

0 

0 

8.4 

2 

31 

0 

0 

0 

1 

4 

12.1 

75 

90 

9.6 

2 

31 

0 

0 

0 

1 

5 

10.6 

30 

0 

7.9 

2 

31 

0 

0 

0 

1 

6 

13.3 

0 

45 

20.1 

2 

31 

0 

0 

1 

1 

1 

3.2 

0 

0 

8.2 

2 

31 

0 

0 

1 

1 

2 

8.1 

30 

0 

6.1 

2 

31 

0 

0 

1 

1 

3 

6.9 

0 

60 

2.2 

2 

31 

0 

0 

1 

1 

4 

7.9 

0 

0 

6.8 

2 

31 

0 

0 

1 

1 

5 

7.7 

0 

0 

6.8 

2 

31 

0 

0 

1 

1 

6 

6.8 

0 

0 

3.4 

3 

60 

0 

0 

0 

0 

1 

10.2 

0 

0 

4.3 

3 

60 

0 

0 

0 

0 

2 

5.2 

45 

0 

7.5 

3 

60 

0 

0 

0 

0 

3 

5 

0 

0 

6.2 

3 

60 

0 

0 

0 

0 

4 

3.8 

0 

165 

8.2 

3 

60 

0 

0 

0 

0 

5 

3.9 

0 

30 

5.3 

3 

60 

0 

0 

0 

0 

6 

3.7 

0 

150 

7.5 

3 

60 

0 

0 

1 

0 

1 

1.9 

0 

0 

4.9 

3 

60 

0 

0 

1 

0 

2 

2.2 

0 

0 

6.5 

3 

60 

0 

0 

1 

0 

3 

2.3 

0 

0 

5.2 

3 

60 

0 

0 

1 

0 

4 

2.3 

0 

0 

5.8 

3 

60 

0 

0 

1 

0 

5 

1.8 

0 

0 

4.7 

3 

60 

0 

0 

1 

0 

6 

1.7 

0 

30 

4 
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Participant 

ID 

Age 

Pilot? 

(0/1) 

Flight 

Hours 

Box? 

(0/1) 

started 

with 

Box? 

(0/1) 

Trial 

Orientation 

Time 

Error 

Pitch 

Error 

Roll 

Recovery 

Time 

4 

23 

0 

0 

0 

1 

1 

3.2 

0 

30 

4.7 

4 

23 

0 

0 

0 

1 

2 

3.1 

15 

90 

19.8 

4 

23 

0 

0 

0 

1 

3 

4.2 

15 

30 

12.3 

4 

23 

0 

0 

0 

1 

4 

7.6 

0 

135 

14 

4 

23 

0 

0 

0 

1 

5 

6.4 

45 

30 

9.2 

4 

23 

0 

0 

0 

1 

6 

6.5 

0 

45 

9.8 

4 

23 

0 

0 

1 

1 

1 

1.8 

0 

45 

8.3 

4 

23 

0 

0 

1 

1 

2 

3 

15 

0 

7.3 

4 

23 

0 

0 

1 

1 

3 

1.7 

0 

0 

7.8 

4 

23 

0 

0 

1 

1 

4 

1.7 

0 

0 

7.7 

4 

23 

0 

0 

1 

1 

5 

1.8 

0 

45 

2.3 

4 

23 

0 

0 

1 

1 

6 

2 

0 

45 

9.8 

5 

28 

0 

0 

0 

0 

1 

9.3 

15 

60 

7.7 

5 

28 

0 

0 

0 

0 

2 

5.3 

0 

45 

12.2 

5 

28 

0 

0 

0 

0 

3 

5.2 

15 

0 

6.5 

5 

28 

0 

0 

0 

0 

4 

6.4 

0 

180 

15.6 

5 

28 

0 

0 

0 

0 

5 

5.1 

0 

180 

7.8 

5 

28 

0 

0 

0 

0 

6 

5.3 

15 

165 

6.7 

5 

28 

0 

0 

1 

0 

1 

1.3 

0 

30 

4.3 

5 

28 

0 

0 

1 

0 

2 

2.6 

0 

15 

8.1 

5 

28 

0 

0 

1 

0 

3 

2.5 

0 

30 

11.8 

5 

28 

0 

0 

1 

0 

4 

3.1 

0 

15 

6.9 

5 

28 

0 

0 

1 

0 

5 

2.8 

0 

30 

3.4 

5 

28 

0 

0 

1 

0 

6 

2.9 

15 

45 

4.1 

101 

34 

1 

500 

0 

0 

1 

3.1 

0 

0 

7 

101 

34 

1 

500 

0 

0 

2 

3.9 

15 

0 

7.3 

101 

34 

1 

500 

0 

0 

3 

3.5 

15 

0 

7.2 

101 

34 

1 

500 

0 

0 

4 

3.6 

15 

0 

10.6 

101 

34 

1 

500 

0 

0 

5 

4.3 

0 

0 

7.7 

101 

34 

1 

500 

0 

0 

6 

3.7 

15 

0 

7.6 

101 

34 

1 

500 

1 

0 

1 

1.4 

0 

0 

6.3 

101 

34 

1 

500 

1 

0 

2 

1.7 

0 

0 

6.2 

101 

34 

1 

500 

1 

0 

3 

1.7 

0 

0 

7.3 

101 

34 

1 

500 

1 

0 

4 

1.7 

0 

0 

5.3 

101 

34 

1 

500 

1 

0 

5 

2.2 

0 

0 

5.1 

101 

34 

1 

500 

1 

0 

6 

1.8 

0 

0 

3.3 

102 

39 

1 

2000 

0 

1 

1 

5.9 

0 

30 

7.8 

102 

39 

1 

2000 

0 

1 

2 

10.3 

0 

0 

6.2 

102 

39 

1 

2000 

0 

1 

3 

7.8 

15 

15 

7.1 
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Participant 

ID 

Age 

Pilot? 

(0/1) 

Flight 

Hours 

Box? 

(0/1) 

started 

with 

Box? 

(0/1) 

Trial 

Orientation 

Time 

Error 

Pitch 

Error 

Roll 

Recovery 

Time 

102 

39 

1 

2000 

0 

1 

4 

8.2 

15 

90 

5.7 

102 

39 

1 

2000 

0 

1 

5 

6.4 

45 

0 

6.3 

102 

39 

1 

2000 

0 

1 

6 

7.8 

0 

180 

5.6 

102 

39 

1 

2000 

1 

1 

1 

4.2 

0 

0 

3.2 

102 

39 

1 

2000 

1 

1 

2 

2.3 

15 

0 

5.7 

102 

39 

1 

2000 

1 

1 

3 

2.8 

15 

0 

3.4 

102 

39 

1 

2000 

1 

1 

4 

2.7 

0 

0 

5.3 

102 

39 

1 

2000 

1 

1 

5 

2.9 

0 

0 

3.1 

102 

39 

1 

2000 

1 

1 

6 

2.7 

0 

0 

5.7 

103 

34 

1 

1100 

0 

0 

1 

8.3 

0 

30 

7.8 

103 

34 

1 

1100 

0 

0 

2 

7.6 

0 

0 

7.6 

103 

34 

1 

1100 

0 

0 

3 

9.6 

15 

30 

7.9 

103 

34 

1 

1100 

0 

0 

4 

6.4 

0 

0 

8.5 

103 

34 

1 

1100 

0 

0 

5 

11.5 

0 

15 

6.8 

103 

34 

1 

1100 

0 

0 

6 

7.5 

15 

0 

9.4 

103 

34 

1 

1100 

1 

0 

1 

2.9 

0 

15 

3.5 

103 

34 

1 

1100 

1 

0 

2 

3.1 

15 

0 

3.4 

103 

34 

1 

1100 

1 

0 

3 

3.2 

0 

0 

2.9 

103 

34 

1 

1100 

1 

0 

4 

3 

0 

15 

3.6 

103 

34 

1 

1100 

1 

0 

5 

3.2 

0 

0 

3 

103 

34 

1 

1100 

1 

0 

6 

3.6 

0 

0 

4.2 

104 

27 

1 

30 

0 

1 

1 

8.9 

0 

90 

14.2 

104 

27 

1 

30 

0 

1 

2 

6.4 

15 

0 

17.6 

104 

27 

1 

30 

0 

1 

3 

6.2 

0 

45 

11.4 

104 

27 

1 

30 

0 

1 

4 

7.2 

0 

15 

12.3 

104 

27 

1 

30 

0 

1 

5 

6.9 

0 

0 

16.1 

104 

27 

1 

30 

0 

1 

6 

6.1 

0 

90 

14.7 

104 

27 

1 

30 

1 

1 

1 

3.7 

0 

15 

10.7 

104 

27 

1 

30 

1 

1 

2 

3.1 

0 

15 

10.4 

104 

27 

1 

30 

1 

1 

3 

4.2 

15 

15 

5.7 

104 

27 

1 

30 

1 

1 

4 

2.9 

0 

45 

7 

104 

27 

1 

30 

1 

1 

5 

1.8 

0 

0 

5 

104 

27 

1 

30 

1 

1 

6 

2.8 

0 

0 

6.7 

6 

32 

0 

0 

0 

1 

1 

3.1 

15 

0 

21.5 

6 

32 

0 

0 

0 

1 

2 

6.9 

30 

30 

7.2 

6 

32 

0 

0 

0 

1 

3 

10.5 

0 

90 

6.8 

6 

32 

0 

0 

0 

1 

4 

7.6 

0 

75 

9.1 

6 

32 

0 

0 

0 

1 

5 

9.2 

0 

9.2 

10.1 

6 

32 

0 

0 

0 

1 

6 

7.6 

15 

7.6 

9.9 
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Participant 

ID 

Age 

Pilot? 

(0/1) 

Flight 

Hours 

Box? 

(0/1) 

started 

with 

Box? 

(0/1) 

Trial 

Orientation 

Time 

Error 

Pitch 

Error 

Roll 

Recovery 

Time 

6 

32 

0 

0 

1 

1 

1 

1.9 

15 

15 

10.2 

6 

32 

0 

0 

1 

1 

2 

1.4 

0 

0 

5.1 

6 

32 

0 

0 

1 

1 

3 

3.1 

15 

15 

10 

6 

32 

0 

0 

1 

1 

4 

1.4 

0 

90 

8.5 

6 

32 

0 

0 

1 

1 

5 

1.7 

0 

0 

11.4 

6 

32 

0 

0 

1 

1 

6 

2.2 

15 

0 

9.8 

7 

31 

0 

0 

0 

0 

1 

15.4 

15 

90 

10.4 

7 

31 

0 

0 

0 

0 

2 

6.8 

0 

90 

8.6 

7 

31 

0 

0 

0 

0 

3 

3.4 

0 

0 

9.2 

7 

31 

0 

0 

0 

0 

4 

4 

0 

0 

8.7 

7 

31 

0 

0 

0 

0 

5 

4.2 

75 

30 

8.9 

7 

31 

0 

0 

0 

0 

6 

4.1 

0 

75 

9.2 

7 

31 

0 

0 

1 

0 

1 

1.7 

0 

15 

4.4 

7 

31 

0 

0 

1 

0 

2 

2.5 

15 

0 

3.9 

7 

31 

0 

0 

1 

0 

3 

1.8 

0 

30 

4.4 

7 

31 

0 

0 

1 

0 

4 

2.1 

0 

0 

3.8 

7 

31 

0 

0 

1 

0 

5 

1.6 

0 

15 

3.9 

7 

31 

0 

0 

1 

0 

6 

2 

0 

0 

2.7 

8 

32 

0 

0 

0 

1 

1 

11.7 

0 

90 

7.3 

8 

32 

0 

0 

0 

1 

2 

10.2 

15 

15 

7.8 

8 

32 

0 

0 

0 

1 

3 

11.1 

0 

0 

8.9 

8 

32 

0 

0 

0 

1 

4 

11.1 

0 

30 

6.7 

8 

32 

0 

0 

0 

1 

5 

10.2 

0 

0 

8.8 

8 

32 

0 

0 

0 

1 

6 

8.1 

45 

15 

11.8 

8 

32 

0 

0 

1 

1 

1 

1.9 

0 

0 

6.8 

8 

32 

0 

0 

1 

1 

2 

1.4 

15 

0 

5.8 

8 

32 

0 

0 

1 

1 

3 

1.8 

15 

15 

6.6 

8 

32 

0 

0 

1 

1 

4 

1.6 

0 

15 

5.3 

8 

32 

0 

0 

1 

1 

5 

2.1 

0 

0 

4.3 

8 

32 

0 

0 

1 

1 

6 

1.9 

0 

0 

7.9 

9 

33 

0 

0 

0 

0 

1 

7.8 

15 

75 

10.8 

9 

33 

0 

0 

0 

0 

2 

4.9 

15 

90 

9.3 

9 

33 

0 

0 

0 

0 

3 

4.1 

15 

0 

9.1 

9 

33 

0 

0 

0 

0 

4 

5.1 

0 

0 

12.2 

9 

33 

0 

0 

0 

0 

5 

5.2 

0 

0 

12.8 

9 

33 

0 

0 

0 

0 

6 

3.8 

0 

0 

15.6 

9 

33 

0 

0 

1 

0 

1 

1.4 

0 

30 

8.9 

9 

33 

0 

0 

1 

0 

2 

1.7 

0 

15 

6.2 

9 

33 

0 

0 

1 

0 

3 

1.3 

0 

30 

3.2 
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Participant 

Age 

ID 


Pilot? 

(0/1) 

Flight 

Hours (o/j) 

started 

with 

Box? 

(0/1) 

Trial 

Orientation 

Time 

Error 

Pitch 

Error 

Roll 

Recovery 

Time 

9 

33 

0 

0 

1 

0 

4 

1.9 

30 

30 

5.7 

9 

33 

0 

0 

1 

0 

5 

1.8 

0 

0 

5.6 

9 

33 

0 

0 

1 

0 

6 

1.9 

0 

15 

5.5 

105 

35 

1 

2500 

0 

0 

1 

3.6 

15 

75 

8.2 

105 

35 

1 

2500 

0 

0 

2 

4.6 

0 

90 

12.9 

105 

35 

1 

2500 

0 

0 

3 

3.7 

30 

0 

7.6 

105 

35 

1 

2500 

0 

0 

4 

3.1 

15 

30 

10 

105 

35 

1 

2500 

0 

0 

5 

3.5 

0 

30 

8.9 

105 

35 

1 

2500 

0 

0 

6 

3.9 

30 

0 

7.8 

105 

35 

1 

2500 

1 

0 

1 

1.7 

0 

30 

4.2 

105 

35 

1 

2500 

1 

0 

2 

2 

0 

0 

4.1 

105 

35 

1 

2500 

1 

0 

3 

2.2 

0 

30 

7.8 

105 

35 

1 

2500 

1 

0 

4 

1.7 

0 

0 

3.3 

105 

35 

1 

2500 

1 

0 

5 

1.9 

0 

0 

4.3 

105 

35 

1 

2500 

1 

0 

6 

1.6 

0 

0 

10.5 

106 

33 

1 

30 

0 

1 

1 

6.8 

0 

135 

6.7 

106 

33 

1 

30 

0 

1 

2 

6.3 

15 

165 

6.3 

106 

33 

1 

30 

0 

1 

3 

6.2 

30 

90 

23 

106 

33 

1 

30 

0 

1 

4 

7.7 

0 

165 

12.4 

106 

33 

1 

30 

0 

1 

5 

6.4 

0 

0 

12.3 

106 

33 

1 

30 

0 

1 

6 

5.9 

0 

90 

7.7 

106 

33 

1 

30 

1 

1 

1 

4.5 

0 

90 

11.1 

106 

33 

1 

30 

1 

1 

2 

2.3 

0 

15 

16 

106 

33 

1 

30 

1 

1 

3 

2.4 

0 

15 

12.8 

106 

33 

1 

30 

1 

1 

4 

2.3 

0 

90 

8.8 

106 

33 

1 

30 

1 

1 

5 

2.4 

0 

15 

8.6 

106 

33 

1 

30 

1 

1 

6 

2.2 

0 

0 

7.5 

107 

36 

1 

1200 

0 

0 

1 

4.1 

0 

90 

13.9 

107 

36 

1 

1200 

0 

0 

2 

6.2 

15 

0 

10.6 

107 

36 

1 

1200 

0 

0 

3 

6.7 

15 

45 

8.9 

107 

36 

1 

1200 

0 

0 

4 

5.4 

0 

90 

16.9 

107 

36 

1 

1200 

0 

0 

5 

6.7 

30 

45 

17.4 

107 

36 

1 

1200 

0 

0 

6 

4.3 

45 

45 

15.7 

107 

36 

1 

1200 

1 

0 

1 

2.8 

0 

30 

4.4 

107 

36 

1 

1200 

1 

0 

2 

2.9 

30 

15 

5.8 

107 

36 

1 

1200 

1 

0 

3 

2.7 

0 

30 

4 

107 

36 

1 

1200 

1 

0 

4 

2.7 

0 

0 

4.5 

107 

36 

1 

1200 

1 

0 

5 

2.1 

0 

30 

5 

107 

36 

1 

1200 

1 

0 

6 

2.2 

0 

0 

5.3 
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Participant 

ID 

Age 

Pilot? 

(0/1) 

Flight ' 

Hours 

Box? 

(0/1) 

started 

with 

Box? 

(0/1) 

Trial 

Orientation 

Time 

Error 

Pitch 

Error 

Roll 

Recovery 

Time 

10 

31 

0 

0 

0 

1 

1 

9.6 

0 

180 

19.2 

10 

31 

0 

0 

0 

1 

2 

6.1 

0 

165 

14.3 

10 

31 

0 

0 

0 

1 

3 

5.8 

0 

30 

15.1 

10 

31 

0 

0 

0 

1 

4 

5.7 

0 

90 

16.1 

10 

31 

0 

0 

0 

1 

5 

5.5 

30 

45 

14.1 

10 

31 

0 

0 

0 

1 

6 

4.8 

0 

90 

11.9 

10 

31 

0 

0 

1 

1 

1 

3.4 

0 

0 

4.6 

10 

31 

0 

0 

1 

1 

2 

3.1 

0 

90 

11 

10 

31 

0 

0 

1 

1 

3 

2.8 

0 

15 

10.1 

10 

31 

0 

0 

1 

1 

4 

3 

0 

0 

9.3 

10 

31 

0 

0 

1 

1 

5 

2.7 

0 

15 

3.9 

10 

31 

0 

0 

1 

1 

6 

3.4 

15 

15 

5.1 

109 

40 

1 

3200 

0 

1 

1 

6.8 

45 

0 

11.8 

109 

40 

1 

3200 

0 

1 

2 

7.8 

90 

0 

10.1 

109 

40 

1 

3200 

0 

1 

3 

8.2 

0 

90 

11.2 

109 

40 

1 

3200 

0 

1 

4 

8.4 

0 

0 

5.2 

109 

40 

1 

3200 

0 

1 

5 

11.8 

0 

15 

13.2 

109 

40 

1 

3200 

0 

1 

6 

9.6 

30 

0 

15.6 

109 

40 

1 

3200 

1 

1 

1 

2.1 

0 

0 

3.3 

109 

40 

1 

3200 

1 

1 

2 

2.3 

0 

0 

3.9 

109 

40 

1 

3200 

1 

1 

3 

1.4 

0 

0 

4.5 

109 

40 

1 

3200 

1 

1 

4 

2.4 

0 

30 

3.3 

109 

40 

1 

3200 

1 

1 

5 

4.3 

0 

30 

5.4 

109 

40 

1 

3200 

1 

1 

6 

3.1 

0 

60 

3.8 

108 

38 

1 

2000 

0 

0 

1 

6.7 

15 

0 

13.1 

108 

38 

1 

2000 

0 

0 

2 

5.8 

0 

90 

6.5 

108 

38 

1 

2000 

0 

0 

3 

7.5 

0 

105 

7.8 

108 

38 

1 

2000 

0 

0 

4 

4.7 

0 

0 

8.1 

108 

38 

1 

2000 

0 

0 

5 

11.8 

0 

0 

7.4 

108 

38 

1 

2000 

0 

0 

6 

7.1 

0 

15 

12.6 

108 

38 

1 

2000 

1 

0 

1 

2.9 

0 

30 

4 

108 

38 

1 

2000 

1 

0 

2 

3 

0 

0 

3 

108 

38 

1 

2000 

1 

0 

3 

3.2 

0 

15 

3.3 

108 

38 

1 

2000 

1 

0 

4 

2.1 

0 

15 

5.5 

108 

38 

1 

2000 

1 

0 

5 

3.2 

0 

0 

4.3 

108 

38 

1 

2000 

1 

0 

6 

1.7 

0 

0 

7.1 

no 

36 

1 

2100 

0 

1 

1 

8.1 

15 

15.0 

12.1 

no 

36 

1 

2100 

0 

1 

2 

11.4 

45 

0.0 

9.6 

no 

36 

1 

2100 

0 

1 

3 

9.7 

60 

45.0 

11.1 


142 




Participant 

ID 

Age 

Pilot? 

(0/1) 

Flight 

Hours 

Box? 

(0/1) 

started 

with 

Box? 

(0/1) 

Orientation Error Error Recovery 

Trial 

Time Pitch Roll Time 

110 

36 

1 

2100 

0 

1 

4 

7.5 

60 

0.0 

8.7 

no 

36 

1 

2100 

0 

1 

5 

15.2 

0 

45.0 

14.9 

no 

36 

1 

2100 

0 

1 

6 

9.8 

45 

0.0 

10.2 

no 

36 

1 

2100 

1 

1 

1 

4.5 

0 

15.0 

4.8 

no 

36 

1 

2100 

1 

1 

2 

4.1 

15 

0.0 

5.0 

no 

36 

1 

2100 

1 

1 

3 

3.5 

0 

0.0 

4.2 

no 

36 

1 

2100 

1 

1 

4 

3.9 

15 

0.0 

5.5 

no 

36 

1 

2100 

1 

1 

5 

3.2 

15 

0.0 

4.1 

no 

36 

1 

2100 

1 

1 

6 

4.2 

0 

15.0 

4.3 


Questionnaire (Orientation Task) 



Task One 

Difficuity 
of Task 

Difficuity of 
SA Task by 
instruments 

Difficuity 
of SA 
Task by 
Box 

Better 

SA with 
Box? 

Box 

Heipfui in 
Monitoring 
Tasks 

Box in- 
/decrease 
Workioad 

Participant 

1-5 

1 -5 

1 -5 

1 -5 

1 -5 

1 -5 

101 

2 

2 

2 

5 

2 

3 

102 

2 

2 

1 

4 

2 

2 

103 

4 

4 

2 

5 

3 

3 

104 

4 

4 

1 

4 

2 

2 

105 

2 

3 

2 

4 

2 

2 

106 

1 

1 

1 

4 

2 

2 

107 

2 

2 

1 

4 

3 

1 

108 

3 

4 

2 

5 

2 

2 

109 

4 

5 

1 

5 

3 

2 

110 

4 

4 

2 

5 

1 

3 

1 

4 

4 

2 

5 

2 

2 

2 

3 

4 

1 

5 

1 

1 

3 

2 

3 

1 

5 

1 

1 

4 

4 

5 

2 

5 

3 

3 

5 

2 

3 

1 

4 

4 

2 

6 

2 

4 

2 

4 

2 

2 

7 

4 

4 

2 

5 

1 

1 

8 

4 

3 

2 

4 

4 

2 

9 

4 

4 

2 

5 

4 

4 

10 

3 

4 

2 

5 

1 

2 
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Q uestionnaire (Recovery Task) 



Task Two 

Difficulty 
of Task 

Difficulty to 
use 

Instruments 

Difficulty 
to use 
Box 

Better 

SA? 

Box 

Helpful in 
Managing 
the 

Aircraft 

Box 

Helpful 

in 

Recovery 

Task 

Box in- 
/decrease 
Monitoring 
Tasks 

Box in- 
/decrease 
Workload 

Participant 

1 -5 

1 -5 

1 -5 

1 -5 

1 -5 

1 -5 

1 -5 

1 -5 

101 

1 

1 

1 

5 

3 

4 

2 

2 

102 

2 

3 

2 

5 

4 

5 

2 

2 

103 

4 

3 

2 

5 

5 

5 

3 

3 

104 

3 

4 

2 

4 

5 

5 

2 

2 

105 

2 

4 

2 

5 

2 

4 

4 

2 

106 

4 

2 

2 

2 

5 

3 

3 

2 

107 

2 

2 

3 

5 

4 

4 

4 

2 

108 

3 

3 

4 

1 

5 

1 

2 

2 

109 

3 

4 

2 

5 

5 

5 

3 

2 

110 

4 

4 

2 

5 

5 

5 

2 

3 

1 

3 

4 

3 

5 

4 

4 

3 

2 

2 

4 

5 

2 

4 

5 

5 

1 

1 

3 

2 

3 

1 

5 

5 

5 

1 

2 

4 

1 

2 

1 

4 

4 

3 

2 

2 

5 

2 

3 

2 

5 

5 

5 

4 

1 

6 

4 

4 

3 

4 

4 

5 

2 

2 

7 

2 

3 

1 

5 

5 

5 

1 

1 

8 

3 

3 

2 

5 

5 

5 

2 

2 

9 

4 

2 

4 

4 

4 

4 

4 

4 

10 

2 

4 

2 

4 

5 

5 

2 

2 
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